[lldb-dev] test rerun phase is in

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Tue Dec 15 14:30:58 PST 2015 

Okay cool.  Will do.

On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen <chying at google.com> wrote:

> Sure. Please go ahead to do that.
> BTW, the pending builds should be merged into one build once current build
> is done.
>
> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Hey Ying,
>>
>> Do you mind if we clear the android builder queue to get a build with
>> r255676 in it?  There are what looks like at least 3 or 4 builds between
>> now and then, and with timeouts it may take several hours.
>>
>> -Todd
>>
>> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen <chying at google.com> wrote:
>>
>>> Yes, it happens every time for android builder.
>>>
>>> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> Hmm, yeah it looks like it did the rerun and then after finishing the
>>>> rerun, it's just hanging.
>>>>
>>>> Let's have a look right after r255676 goes through this builder.  I hit
>>>> a hang in the curses output display due to the recursive taking of a lock
>>>> on a lock that was not recursive-enabled.  While I would have expected to
>>>> see that with the basic results output that this builder here is using when
>>>> I was testing earlier, it's possible somehow that we're hitting a path here
>>>> that is attempting to recursively take a lock.
>>>>
>>>> Do you know if it is happening every single time a rerun occurs?
>>>>  (Hopefully yes?)
>>>>
>>>> -Todd
>>>>
>>>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> Yep, I'll have a look!
>>>>>
>>>>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen <chying at google.com> wrote:
>>>>>
>>>>>> Hi Todd,
>>>>>>
>>>>>> It is noticed on lldb android builders that the test_runner didn't
>>>>>> exit after rerun, which caused buildbot timeout since the process was
>>>>>> hanging for over 20 minutes.
>>>>>> Could you please take a look if that's related to your change?
>>>>>>
>>>>>> Please see the following builds.
>>>>>>
>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>>>>>
>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>>>>>
>>>>>> Thanks,
>>>>>> Ying
>>>>>>
>>>>>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>
>>>>>>> And, btw, this shows the rerun logic working (via the
>>>>>>> --rerun-all-issues flag):
>>>>>>>
>>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
>>>>>>> --rerun-all-issues
>>>>>>> Testing: 416 test suites, 24 threads
>>>>>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>>>>>>
>>>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>>>> into directory '2015-12-14-16_44_28'
>>>>>>> Command invoked: test/dotest.py --executable
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>> -p TestMultithreaded.py
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>> --event-add-entries worker_index=3:int
>>>>>>>
>>>>>>> Configuration: arch=x86_64 compiler=clang
>>>>>>>
>>>>>>> ----------------------------------------------------------------------
>>>>>>> Collected 8 tests
>>>>>>>
>>>>>>> lldb_codesign: no identity found
>>>>>>> lldb_codesign: no identity found
>>>>>>> lldb_codesign: no identity found
>>>>>>> lldb_codesign: no identity found
>>>>>>> lldb_codesign: no identity found
>>>>>>> lldb_codesign: no identity found
>>>>>>> lldb_codesign: no identity found
>>>>>>>
>>>>>>> [TestMultithreaded.py FAILED]
>>>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>> -p TestMultithreaded.py
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>> --event-add-entries worker_index=3:int
>>>>>>> 396 out of 416 test suites processed - TestMiBreak.py
>>>>>>>
>>>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>>>> into directory '2015-12-14-16_44_28'
>>>>>>> Command invoked: test/dotest.py --executable
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>> -p TestDataFormatterObjC.py
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>> --event-add-entries worker_index=12:int
>>>>>>>
>>>>>>> Configuration: arch=x86_64 compiler=clang
>>>>>>>
>>>>>>> ----------------------------------------------------------------------
>>>>>>> Collected 26 tests
>>>>>>>
>>>>>>>
>>>>>>> [TestDataFormatterObjC.py FAILED]
>>>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>> -p TestDataFormatterObjC.py
>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>> --event-add-entries worker_index=12:int
>>>>>>> 416 out of 416 test suites processed - TestLldbGdbServer.py
>>>>>>> 2 test files marked for rerun
>>>>>>>
>>>>>>>
>>>>>>> Rerunning the following files:
>>>>>>>
>>>>>>> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>>>>>>>   api/multithreaded/TestMultithreaded.py
>>>>>>> Testing: 2 test suites, 1 thread
>>>>>>> 2 out of 2 test suites processed - TestMultithreaded.py
>>>>>>> Test rerun complete
>>>>>>>
>>>>>>>
>>>>>>> =============
>>>>>>> Issue Details
>>>>>>> =============
>>>>>>> UNEXPECTED SUCCESS: test_symbol_name_dsym
>>>>>>> (functionalities/completion/TestCompletion.py)
>>>>>>> UNEXPECTED SUCCESS: test_symbol_name_dwarf
>>>>>>> (functionalities/completion/TestCompletion.py)
>>>>>>>
>>>>>>> ===================
>>>>>>> Test Result Summary
>>>>>>> ===================
>>>>>>> Test Methods:       1695
>>>>>>> Reruns:               30
>>>>>>> Success:            1367
>>>>>>> Expected Failure:     90
>>>>>>> Failure:               0
>>>>>>> Error:                 0
>>>>>>> Exceptional Exit:      0
>>>>>>> Unexpected Success:    2
>>>>>>> Skip:                236
>>>>>>> Timeout:               0
>>>>>>> Expected Timeout:      0
>>>>>>>
>>>>>>> On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> And that fixed the rest as well.  Thanks, Siva!
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Heh you were skinning the same cat :-)
>>>>>>>>>
>>>>>>>>> That fixed the one I was just looking at, running the others now.
>>>>>>>>>
>>>>>>>>> On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yep, will try now...  (I was just looking at the condition
>>>>>>>>>> testing logic since it looks like something isn't quite right there).
>>>>>>>>>>
>>>>>>>>>> On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra <
>>>>>>>>>> sivachandra at google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can you try again after taking my change at r255584?
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev
>>>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>>> > I'm having some of these blow up.
>>>>>>>>>>> >
>>>>>>>>>>> > In the case of test/lang/c/typedef/Testtypedef.py, it looks
>>>>>>>>>>> like some of the
>>>>>>>>>>> > @expected decorators were changed a bit, and perhaps they are
>>>>>>>>>>> not pound for
>>>>>>>>>>> > pound the same.  For example, this test used to really be
>>>>>>>>>>> marked XFAIL (via
>>>>>>>>>>> > an expectedFailureClang directive), but it looks like the
>>>>>>>>>>> current marking of
>>>>>>>>>>> > compiler="clang" is either not right or not working, since the
>>>>>>>>>>> test is run
>>>>>>>>>>> > on OS X and is treated like it is expected to pass.
>>>>>>>>>>> >
>>>>>>>>>>> > I'm drilling into that a bit more, that's just the first of
>>>>>>>>>>> several that
>>>>>>>>>>> > fail with these changes on OS X.
>>>>>>>>>>> >
>>>>>>>>>>> > On Mon, Dec 14, 2015 at 3:03 PM, Zachary Turner <
>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>> >>
>>>>>>>>>>> >> I've checked in r255567 which fixes a problem pointed out by
>>>>>>>>>>> Siva.  It
>>>>>>>>>>> >> doesn't sound related to in 255542, but looking at those logs
>>>>>>>>>>> I can't really
>>>>>>>>>>> >> tell how my CL would be related.  If r255567 doesn't fix the
>>>>>>>>>>> bots, would
>>>>>>>>>>> >> someone mind helping me briefly?  r255542 seems pretty
>>>>>>>>>>> straightforward, so I
>>>>>>>>>>> >> don't see why it would have an effect here.
>>>>>>>>>>> >>
>>>>>>>>>>> >> On Mon, Dec 14, 2015 at 2:35 PM Todd Fiala <
>>>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> Ah yes I see.  Thanks, Ying (and Siva!  Saw your comments
>>>>>>>>>>> too).
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> On Mon, Dec 14, 2015 at 2:34 PM, Ying Chen <
>>>>>>>>>>> chying at google.com> wrote:
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> Seems this is the first build that fails, and it only has
>>>>>>>>>>> one CL 255542.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446
>>>>>>>>>>> >>>> I believe Zachary is looking at that problem.
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>> On Mon, Dec 14, 2015 at 2:18 PM, Todd Fiala <
>>>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>>>> >>>> wrote:
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> I am seeing several failures on the Ubuntu 14.04 testbot,
>>>>>>>>>>> but
>>>>>>>>>>> >>>>> unfortunately there are a number of changes that went in
>>>>>>>>>>> at the same time on
>>>>>>>>>>> >>>>> that build.  The failures I'm seeing are not appearing at
>>>>>>>>>>> all related to the
>>>>>>>>>>> >>>>> test running infrastructure.
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> Anybody with a fast Linux system able to take a look to
>>>>>>>>>>> see what
>>>>>>>>>>> >>>>> exactly is failing there?
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> -Todd
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> On Mon, Dec 14, 2015 at 1:39 PM, Todd Fiala <
>>>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>>>> >>>>> wrote:
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> Hi all,
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> I just put in the single-worker, low-load, follow-up test
>>>>>>>>>>> run pass in
>>>>>>>>>>> >>>>>> r255543.  Most of the work for it went in late last week,
>>>>>>>>>>> this just mostly
>>>>>>>>>>> >>>>>> flips it on.
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> The feature works like this:
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> * First test phase works as before: run all tests using
>>>>>>>>>>> whatever level
>>>>>>>>>>> >>>>>> of concurrency is normally used.  (e.g. 8 works on an
>>>>>>>>>>> 8-logical-core box).
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> * Any timeouts, failures, errors, or anything else that
>>>>>>>>>>> would have
>>>>>>>>>>> >>>>>> caused a test failure is eligible for rerun if either (1)
>>>>>>>>>>> it was marked as a
>>>>>>>>>>> >>>>>> flakey test via the flakey decorator, or (2) if the
>>>>>>>>>>> --rerun-all-issues
>>>>>>>>>>> >>>>>> command line flag is provided.
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> * After the first test phase, if there are any tests that
>>>>>>>>>>> met rerun
>>>>>>>>>>> >>>>>> eligibility that would have caused a test failure, those
>>>>>>>>>>> get run using a
>>>>>>>>>>> >>>>>> serial test phase.  Their results will overwrite (i.e.
>>>>>>>>>>> replace) the previous
>>>>>>>>>>> >>>>>> result for the given test method.
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> The net result should be that tests that were load
>>>>>>>>>>> sensitive and
>>>>>>>>>>> >>>>>> intermittently fail during the first higher-concurrency
>>>>>>>>>>> test phase should
>>>>>>>>>>> >>>>>> (in theory) pass in the second, single worker test phase
>>>>>>>>>>> when the test suite
>>>>>>>>>>> >>>>>> is only using a single worker.  This should make the test
>>>>>>>>>>> suite generate
>>>>>>>>>>> >>>>>> fewer false positives on test failure notification, which
>>>>>>>>>>> should make
>>>>>>>>>>> >>>>>> continuous integration servers (testbots) much more
>>>>>>>>>>> useful in terms of
>>>>>>>>>>> >>>>>> generating actionable signals caused by version control
>>>>>>>>>>> changes to the lldb
>>>>>>>>>>> >>>>>> or related sources.
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> Please let me know if you see any issues with this when
>>>>>>>>>>> running the
>>>>>>>>>>> >>>>>> test suite using the default output.  I'd like to fix
>>>>>>>>>>> this up ASAP.  And for
>>>>>>>>>>> >>>>>> those interested in the implementation, I'm happy to do
>>>>>>>>>>> post-commit
>>>>>>>>>>> >>>>>> review/changes as needed to get it in good shape.
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> I'll be watching the  builders now and will address any
>>>>>>>>>>> issues as I
>>>>>>>>>>> >>>>>> see them.
>>>>>>>>>>> >>>>>>
>>>>>>>>>>> >>>>>> Thanks!
>>>>>>>>>>> >>>>>> --
>>>>>>>>>>> >>>>>> -Todd
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>>
>>>>>>>>>>> >>>>> --
>>>>>>>>>>> >>>>> -Todd
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>>
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>
>>>>>>>>>>> >>>
>>>>>>>>>>> >>> --
>>>>>>>>>>> >>> -Todd
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > -Todd
>>>>>>>>>>> >
>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>> > lldb-dev mailing list
>>>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> lldb-dev mailing list
>>>>>>> lldb-dev at lists.llvm.org
>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151215/345a9d59/attachment-0001.html>

More information about the lldb-dev mailing list