[Lldb-commits] [PATCH] 5 minute timeout for tests

[Vince Harron]

Currently when the tests lock up on the build server, the build script
kills the tests and gets *no* test results.  That means that if a bug goes
in that causes a test to hang, we don't get results for *any* tests.

Killing the individual test causes it to show up as a FAIL (Chaoren might
change this to show result as "TIMEOUT")

It's not really about what tests are hanging today, it's that some day a
test will hang.  When that happens, do we want to lose all test results as
a result?  Look at the Linux buildbot history.  Many runs have no test
results at all.  Which test caused the lockup?  I have no idea.

> Both of those options are really undesirable in my opinion.

We can make the test timeout be an OSX/Linux thing only for now...

> Regardless, I kind of don't like the idea of making it easier to ignore
broken tests.  If it's that painful, I think the test should be either
fixed or disabled.

If a timeout test shows up as a failure, I don't think you're ignoring it.


On Mon, Dec 1, 2014 at 2:48 PM, Zachary Turner <zturner at google.com> wrote:

> Note: At some point lldb-commits got lost from the Reply list.  I'm
> putting it back just so that other interested parties aren't kept out of
> the loop.
>
> --
>
> I'm kind of against this unless we know why the tests are failing, and
> maybe even then.  Is this really a common enough thing that we need to
> either increase the maintenance burden of the test suite by adding features
> to it (any features, much less ones that are implemented differently on
> each platform), or have to add a dependency on a 3rd party python module?
> Both of those options are really undesirable in my opinion.
>
> As far as I can tell, this is just for running the test suite locally --
> the bots already have a global timeout.  If a test hangs locally, you can
> just kill it with a shell command in less time than it would take for this
> 5 minute timeout to expire.
>
> Regardless, I kind of don't like the idea of making it easier to ignore
> broken tests.  If it's that painful, I think the test should be either
> fixed or disabled.
>
> On Mon Dec 01 2014 at 2:27:40 PM Chaoren Lin <chaorenl at google.com> wrote:
>
>> Why are the tests hanging?
>>
>>
>> No clue.
>>
>>
>>> Can we just disable the hanging tests?
>>
>>
>> We still want them to run, they should just be equivalent to failing
>> tests.
>>
>>
>> On Mon, Dec 1, 2014 at 10:06 AM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> If it does go in, i think i would actually prefer psutil over platform
>>> specific implementations of it.
>>>
>>> On Mon, Dec 1, 2014 at 10:00 AM Zachary Turner <zturner at google.com>
>>> wrote:
>>>
>>>> Why are the tests hanging?  Can we just disable the hanging tests?  I'm
>>>> a little leary of sinking more platform specific logic into the test
>>>> runner.
>>>>
>>>> On Mon Dec 01 2014 at 9:56:52 AM Chaoren Lin <chaorenl at google.com>
>>>> wrote:
>>>>
>>>>> Thanks for the link Zachary, but unfortunately, we need to kill the
>>>>> entire process tree, and I don't think there's a way to do that portably in
>>>>> python itself. There's psutil, but we probably shouldn't add another
>>>>> dependency just for this convenience.
>>>>>
>>>>> I think timeout works really well for linux, would it be a terrible
>>>>> idea to use that only for linux, TASKKILL /T for windows, and then
>>>>> something else for OS X?
>>>>>
>>>>>
>>>>> On Fri Nov 21 2014 at 4:20:59 PM Zachary Turner <zturner at google.com>
>>>>> wrote:
>>>>>
>>>>>> Maybe this is a good starting point:
>>>>>>
>>>>>> http://stackoverflow.com/questions/1191374/subprocess-with-timeout
>>>>>>
>>>>>> http://reviews.llvm.org/D6364
>>>>>>
>>>>>>
>>>>>>
>>


-- 

Vince Harron | Technical Lead Manager | vharron at google.com | 858-442-0868
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-commits/attachments/20141201/de4d19b4/attachment.html>