[lldb-dev] increase timeout for tests?

[Jim Ingham via lldb-dev]

It sounds like we timing out based on the whole test class, not the individual tests?  If you're worried about test failures not hanging up the test suite the you really want to do the latter.

These are all tests that contain 5 or more independent tests.  That's probably why they are taking so long to run.  

I don't object to having fairly long backstop timeouts, though I agree with Pavel that we should choose something reasonable based on the slowest running tests just so some single error doesn't cause test runs to just never complete, making analysis harder.

Jim

> On Mar 13, 2018, at 10:43 AM, Davide Italiano <dccitaliano at gmail.com> wrote:
> 
> Few examples:
> 
> 360 out of 617 test suites processed - TestObjCMethods.py
>                         XXXXXXXXXXXXXXXXXX
> TestObjCMethods.py: 363.726592
> 381 out of 617 test suites processed - TestHiddenIvars.py
>                            XXXXXXXXXXXXXXXXXX
>  TestHiddenIvars.py: 363.887766
> 387 out of 617 test suites processed - TestObjCNewSyntax.py
>                       XXXXXXXXXXXXXXXXXX
> TestObjCNewSyntax.py: 363.842807
> 600 out of 617 test suites processed - TestDataFormatterObjC.py
>                     XXXXXXXXXXXXXXXXXX
> TestDataFormatterObjC.py: 388.414883
> 617 out of 617 test suites processed - TestLoadUnload.py
>                          XXXXXXXXXXXXXXXXXX
> TestLoadUnload.py: 363.372328
> 
> On Tue, Mar 13, 2018 at 9:49 AM, Davide Italiano <dccitaliano at gmail.com> wrote:
>> I'll re-run the test and send you a list.
>> 
>> Thank you!
>> 
>> --
>> Davide
>> 
>> On Tue, Mar 13, 2018 at 9:02 AM, Pavel Labath <labath at google.com> wrote:
>>> Aha, in that case, it definitely sounds like increasing the timeout is in
>>> order. I would still go for something less than 30 minutes, but I don't care
>>> about that much. I may just export LLDB_TEST_TIMEOUT locally to lower it if
>>> tests taking long to time out start bugging me.
>>> 
>>> BTW, do you know which test is that? Is it one of the tests I have listed in
>>> one of the previous emails?
>>> 
>>> On Tue, 13 Mar 2018 at 14:52, Davide Italiano <dccitaliano at gmail.com> wrote:
>>>> 
>>>> On Tue, Mar 13, 2018 at 3:30 AM, Pavel Labath <labath at google.com> wrote:
>>>>> I think we should get some data before pulling numbers out of our
>>>>> sleeves.
>>>>> If we can get some numbers on the slowest machine that we have around,
>>>>> then
>>>>> we should be able to specify the timeout as some multiple of the slowest
>>>>> test. For example, for me the slowest test takes around 110 seconds. The
>>>>> timeout is 4 minutes (~ 2x) and I don't get tests timing out. How long
>>>>> does
>>>>> the slowest test take for you? If we set the timeout as 3x or 4x that
>>>>> number, we should create a sufficiently large buffer and still avoid
>>>>> half-hour waits.
>>>>> 
>>>> 
>>>> The longest test takes over 300 seconds. This is a late 2013 Mac Pro,
>>>> so, definitely not the newest machine out there (but also something
>>>> fairly high spec).
>>>> For the archives, my conf is something like; cmake -GNinja ../llvm &&
>>>> ninja check-lldb
>>>> 
>>>> Thanks!
>>>> 
>>>> --
>>>> Davide