[lldb-dev] test rerun phase is in

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Tue Dec 15 14:12:52 PST 2015 

Hey Ying,

Do you mind if we clear the android builder queue to get a build with
r255676 in it?  There are what looks like at least 3 or 4 builds between
now and then, and with timeouts it may take several hours.

-Todd

On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen <chying at google.com> wrote:

> Yes, it happens every time for android builder.
>
> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Hmm, yeah it looks like it did the rerun and then after finishing the
>> rerun, it's just hanging.
>>
>> Let's have a look right after r255676 goes through this builder.  I hit a
>> hang in the curses output display due to the recursive taking of a lock on
>> a lock that was not recursive-enabled.  While I would have expected to see
>> that with the basic results output that this builder here is using when I
>> was testing earlier, it's possible somehow that we're hitting a path here
>> that is attempting to recursively take a lock.
>>
>> Do you know if it is happening every single time a rerun occurs?
>>  (Hopefully yes?)
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> Yep, I'll have a look!
>>>
>>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen <chying at google.com> wrote:
>>>
>>>> Hi Todd,
>>>>
>>>> It is noticed on lldb android builders that the test_runner didn't exit
>>>> after rerun, which caused buildbot timeout since the process was hanging
>>>> for over 20 minutes.
>>>> Could you please take a look if that's related to your change?
>>>>
>>>> Please see the following builds.
>>>>
>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>>>
>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>>>
>>>> Thanks,
>>>> Ying
>>>>
>>>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>>>> lldb-dev at lists.llvm.org> wrote:
>>>>
>>>>> And, btw, this shows the rerun logic working (via the
>>>>> --rerun-all-issues flag):
>>>>>
>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
>>>>> --rerun-all-issues
>>>>> Testing: 416 test suites, 24 threads
>>>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>>>>
>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>> into directory '2015-12-14-16_44_28'
>>>>> Command invoked: test/dotest.py --executable
>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>> -p TestMultithreaded.py
>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>> --event-add-entries worker_index=3:int
>>>>>
>>>>> Configuration: arch=x86_64 compiler=clang
>>>>> ----------------------------------------------------------------------
>>>>> Collected 8 tests
>>>>>
>>>>> lldb_codesign: no identity found
>>>>> lldb_codesign: no identity found
>>>>> lldb_codesign: no identity found
>>>>> lldb_codesign: no identity found
>>>>> lldb_codesign: no identity found
>>>>> lldb_codesign: no identity found
>>>>> lldb_codesign: no identity found
>>>>>
>>>>> [TestMultithreaded.py FAILED]
>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>> -p TestMultithreaded.py
>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>> --event-add-entries worker_index=3:int
>>>>> 396 out of 416 test suites processed - TestMiBreak.py
>>>>>
>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>> into directory '2015-12-14-16_44_28'
>>>>> Command invoked: test/dotest.py --executable
>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>> -p TestDataFormatterObjC.py
>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>> --event-add-entries worker_index=12:int
>>>>>
>>>>> Configuration: arch=x86_64 compiler=clang
>>>>> ----------------------------------------------------------------------
>>>>> Collected 26 tests
>>>>>
>>>>>
>>>>> [TestDataFormatterObjC.py FAILED]
>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>> -p TestDataFormatterObjC.py
>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>> --event-add-entries worker_index=12:int
>>>>> 416 out of 416 test suites processed - TestLldbGdbServer.py
>>>>> 2 test files marked for rerun
>>>>>
>>>>>
>>>>> Rerunning the following files:
>>>>>
>>>>> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>>>>>   api/multithreaded/TestMultithreaded.py
>>>>> Testing: 2 test suites, 1 thread
>>>>> 2 out of 2 test suites processed - TestMultithreaded.py
>>>>> Test rerun complete
>>>>>
>>>>>
>>>>> =============
>>>>> Issue Details
>>>>> =============
>>>>> UNEXPECTED SUCCESS: test_symbol_name_dsym
>>>>> (functionalities/completion/TestCompletion.py)
>>>>> UNEXPECTED SUCCESS: test_symbol_name_dwarf
>>>>> (functionalities/completion/TestCompletion.py)
>>>>>
>>>>> ===================
>>>>> Test Result Summary
>>>>> ===================
>>>>> Test Methods:       1695
>>>>> Reruns:               30
>>>>> Success:            1367
>>>>> Expected Failure:     90
>>>>> Failure:               0
>>>>> Error:                 0
>>>>> Exceptional Exit:      0
>>>>> Unexpected Success:    2
>>>>> Skip:                236
>>>>> Timeout:               0
>>>>> Expected Timeout:      0
>>>>>
>>>>> On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> And that fixed the rest as well.  Thanks, Siva!
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Heh you were skinning the same cat :-)
>>>>>>>
>>>>>>> That fixed the one I was just looking at, running the others now.
>>>>>>>
>>>>>>> On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yep, will try now...  (I was just looking at the condition testing
>>>>>>>> logic since it looks like something isn't quite right there).
>>>>>>>>
>>>>>>>> On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra <
>>>>>>>> sivachandra at google.com> wrote:
>>>>>>>>
>>>>>>>>> Can you try again after taking my change at r255584?
>>>>>>>>>
>>>>>>>>> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev
>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>> > I'm having some of these blow up.
>>>>>>>>> >
>>>>>>>>> > In the case of test/lang/c/typedef/Testtypedef.py, it looks like
>>>>>>>>> some of the
>>>>>>>>> > @expected decorators were changed a bit, and perhaps they are
>>>>>>>>> not pound for
>>>>>>>>> > pound the same.  For example, this test used to really be marked
>>>>>>>>> XFAIL (via
>>>>>>>>> > an expectedFailureClang directive), but it looks like the
>>>>>>>>> current marking of
>>>>>>>>> > compiler="clang" is either not right or not working, since the
>>>>>>>>> test is run
>>>>>>>>> > on OS X and is treated like it is expected to pass.
>>>>>>>>> >
>>>>>>>>> > I'm drilling into that a bit more, that's just the first of
>>>>>>>>> several that
>>>>>>>>> > fail with these changes on OS X.
>>>>>>>>> >
>>>>>>>>> > On Mon, Dec 14, 2015 at 3:03 PM, Zachary Turner <
>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>> >>
>>>>>>>>> >> I've checked in r255567 which fixes a problem pointed out by
>>>>>>>>> Siva.  It
>>>>>>>>> >> doesn't sound related to in 255542, but looking at those logs I
>>>>>>>>> can't really
>>>>>>>>> >> tell how my CL would be related.  If r255567 doesn't fix the
>>>>>>>>> bots, would
>>>>>>>>> >> someone mind helping me briefly?  r255542 seems pretty
>>>>>>>>> straightforward, so I
>>>>>>>>> >> don't see why it would have an effect here.
>>>>>>>>> >>
>>>>>>>>> >> On Mon, Dec 14, 2015 at 2:35 PM Todd Fiala <
>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> Ah yes I see.  Thanks, Ying (and Siva!  Saw your comments too).
>>>>>>>>> >>>
>>>>>>>>> >>> On Mon, Dec 14, 2015 at 2:34 PM, Ying Chen <chying at google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> Seems this is the first build that fails, and it only has one
>>>>>>>>> CL 255542.
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446
>>>>>>>>> >>>> I believe Zachary is looking at that problem.
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Mon, Dec 14, 2015 at 2:18 PM, Todd Fiala <
>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>> >>>> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> I am seeing several failures on the Ubuntu 14.04 testbot, but
>>>>>>>>> >>>>> unfortunately there are a number of changes that went in at
>>>>>>>>> the same time on
>>>>>>>>> >>>>> that build.  The failures I'm seeing are not appearing at
>>>>>>>>> all related to the
>>>>>>>>> >>>>> test running infrastructure.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> Anybody with a fast Linux system able to take a look to see
>>>>>>>>> what
>>>>>>>>> >>>>> exactly is failing there?
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> -Todd
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Mon, Dec 14, 2015 at 1:39 PM, Todd Fiala <
>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>> >>>>> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Hi all,
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> I just put in the single-worker, low-load, follow-up test
>>>>>>>>> run pass in
>>>>>>>>> >>>>>> r255543.  Most of the work for it went in late last week,
>>>>>>>>> this just mostly
>>>>>>>>> >>>>>> flips it on.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> The feature works like this:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> * First test phase works as before: run all tests using
>>>>>>>>> whatever level
>>>>>>>>> >>>>>> of concurrency is normally used.  (e.g. 8 works on an
>>>>>>>>> 8-logical-core box).
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> * Any timeouts, failures, errors, or anything else that
>>>>>>>>> would have
>>>>>>>>> >>>>>> caused a test failure is eligible for rerun if either (1)
>>>>>>>>> it was marked as a
>>>>>>>>> >>>>>> flakey test via the flakey decorator, or (2) if the
>>>>>>>>> --rerun-all-issues
>>>>>>>>> >>>>>> command line flag is provided.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> * After the first test phase, if there are any tests that
>>>>>>>>> met rerun
>>>>>>>>> >>>>>> eligibility that would have caused a test failure, those
>>>>>>>>> get run using a
>>>>>>>>> >>>>>> serial test phase.  Their results will overwrite (i.e.
>>>>>>>>> replace) the previous
>>>>>>>>> >>>>>> result for the given test method.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> The net result should be that tests that were load
>>>>>>>>> sensitive and
>>>>>>>>> >>>>>> intermittently fail during the first higher-concurrency
>>>>>>>>> test phase should
>>>>>>>>> >>>>>> (in theory) pass in the second, single worker test phase
>>>>>>>>> when the test suite
>>>>>>>>> >>>>>> is only using a single worker.  This should make the test
>>>>>>>>> suite generate
>>>>>>>>> >>>>>> fewer false positives on test failure notification, which
>>>>>>>>> should make
>>>>>>>>> >>>>>> continuous integration servers (testbots) much more useful
>>>>>>>>> in terms of
>>>>>>>>> >>>>>> generating actionable signals caused by version control
>>>>>>>>> changes to the lldb
>>>>>>>>> >>>>>> or related sources.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Please let me know if you see any issues with this when
>>>>>>>>> running the
>>>>>>>>> >>>>>> test suite using the default output.  I'd like to fix this
>>>>>>>>> up ASAP.  And for
>>>>>>>>> >>>>>> those interested in the implementation, I'm happy to do
>>>>>>>>> post-commit
>>>>>>>>> >>>>>> review/changes as needed to get it in good shape.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> I'll be watching the  builders now and will address any
>>>>>>>>> issues as I
>>>>>>>>> >>>>>> see them.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Thanks!
>>>>>>>>> >>>>>> --
>>>>>>>>> >>>>>> -Todd
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> --
>>>>>>>>> >>>>> -Todd
>>>>>>>>> >>>>
>>>>>>>>> >>>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>>
>>>>>>>>> >>> --
>>>>>>>>> >>> -Todd
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > -Todd
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > lldb-dev mailing list
>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>> _______________________________________________
>>>>> lldb-dev mailing list
>>>>> lldb-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151215/5779bb36/attachment-0001.html>

More information about the lldb-dev mailing list