[lldb-dev] test rerun phase is in

Ying Chen via lldb-dev lldb-dev at lists.llvm.org
Tue Dec 15 14:22:37 PST 2015 

Sure. Please go ahead to do that.
BTW, the pending builds should be merged into one build once current build
is done.

On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Hey Ying,
>
> Do you mind if we clear the android builder queue to get a build with
> r255676 in it?  There are what looks like at least 3 or 4 builds between
> now and then, and with timeouts it may take several hours.
>
> -Todd
>
> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen <chying at google.com> wrote:
>
>> Yes, it happens every time for android builder.
>>
>> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> Hmm, yeah it looks like it did the rerun and then after finishing the
>>> rerun, it's just hanging.
>>>
>>> Let's have a look right after r255676 goes through this builder.  I hit
>>> a hang in the curses output display due to the recursive taking of a lock
>>> on a lock that was not recursive-enabled.  While I would have expected to
>>> see that with the basic results output that this builder here is using when
>>> I was testing earlier, it's possible somehow that we're hitting a path here
>>> that is attempting to recursively take a lock.
>>>
>>> Do you know if it is happening every single time a rerun occurs?
>>>  (Hopefully yes?)
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> Yep, I'll have a look!
>>>>
>>>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen <chying at google.com> wrote:
>>>>
>>>>> Hi Todd,
>>>>>
>>>>> It is noticed on lldb android builders that the test_runner didn't
>>>>> exit after rerun, which caused buildbot timeout since the process was
>>>>> hanging for over 20 minutes.
>>>>> Could you please take a look if that's related to your change?
>>>>>
>>>>> Please see the following builds.
>>>>>
>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>>>>
>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>>>>
>>>>> Thanks,
>>>>> Ying
>>>>>
>>>>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>
>>>>>> And, btw, this shows the rerun logic working (via the
>>>>>> --rerun-all-issues flag):
>>>>>>
>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
>>>>>> --rerun-all-issues
>>>>>> Testing: 416 test suites, 24 threads
>>>>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>>>>>
>>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>>> into directory '2015-12-14-16_44_28'
>>>>>> Command invoked: test/dotest.py --executable
>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>> -p TestMultithreaded.py
>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>> --event-add-entries worker_index=3:int
>>>>>>
>>>>>> Configuration: arch=x86_64 compiler=clang
>>>>>> ----------------------------------------------------------------------
>>>>>> Collected 8 tests
>>>>>>
>>>>>> lldb_codesign: no identity found
>>>>>> lldb_codesign: no identity found
>>>>>> lldb_codesign: no identity found
>>>>>> lldb_codesign: no identity found
>>>>>> lldb_codesign: no identity found
>>>>>> lldb_codesign: no identity found
>>>>>> lldb_codesign: no identity found
>>>>>>
>>>>>> [TestMultithreaded.py FAILED]
>>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>> -p TestMultithreaded.py
>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>> --event-add-entries worker_index=3:int
>>>>>> 396 out of 416 test suites processed - TestMiBreak.py
>>>>>>
>>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>>> into directory '2015-12-14-16_44_28'
>>>>>> Command invoked: test/dotest.py --executable
>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>> -p TestDataFormatterObjC.py
>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>> --event-add-entries worker_index=12:int
>>>>>>
>>>>>> Configuration: arch=x86_64 compiler=clang
>>>>>> ----------------------------------------------------------------------
>>>>>> Collected 26 tests
>>>>>>
>>>>>>
>>>>>> [TestDataFormatterObjC.py FAILED]
>>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>> -p TestDataFormatterObjC.py
>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>> --event-add-entries worker_index=12:int
>>>>>> 416 out of 416 test suites processed - TestLldbGdbServer.py
>>>>>> 2 test files marked for rerun
>>>>>>
>>>>>>
>>>>>> Rerunning the following files:
>>>>>>
>>>>>> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>>>>>>   api/multithreaded/TestMultithreaded.py
>>>>>> Testing: 2 test suites, 1 thread
>>>>>> 2 out of 2 test suites processed - TestMultithreaded.py
>>>>>> Test rerun complete
>>>>>>
>>>>>>
>>>>>> =============
>>>>>> Issue Details
>>>>>> =============
>>>>>> UNEXPECTED SUCCESS: test_symbol_name_dsym
>>>>>> (functionalities/completion/TestCompletion.py)
>>>>>> UNEXPECTED SUCCESS: test_symbol_name_dwarf
>>>>>> (functionalities/completion/TestCompletion.py)
>>>>>>
>>>>>> ===================
>>>>>> Test Result Summary
>>>>>> ===================
>>>>>> Test Methods:       1695
>>>>>> Reruns:               30
>>>>>> Success:            1367
>>>>>> Expected Failure:     90
>>>>>> Failure:               0
>>>>>> Error:                 0
>>>>>> Exceptional Exit:      0
>>>>>> Unexpected Success:    2
>>>>>> Skip:                236
>>>>>> Timeout:               0
>>>>>> Expected Timeout:      0
>>>>>>
>>>>>> On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> And that fixed the rest as well.  Thanks, Siva!
>>>>>>>
>>>>>>> -Todd
>>>>>>>
>>>>>>> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Heh you were skinning the same cat :-)
>>>>>>>>
>>>>>>>> That fixed the one I was just looking at, running the others now.
>>>>>>>>
>>>>>>>> On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yep, will try now...  (I was just looking at the condition testing
>>>>>>>>> logic since it looks like something isn't quite right there).
>>>>>>>>>
>>>>>>>>> On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra <
>>>>>>>>> sivachandra at google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Can you try again after taking my change at r255584?
>>>>>>>>>>
>>>>>>>>>> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev
>>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>> > I'm having some of these blow up.
>>>>>>>>>> >
>>>>>>>>>> > In the case of test/lang/c/typedef/Testtypedef.py, it looks
>>>>>>>>>> like some of the
>>>>>>>>>> > @expected decorators were changed a bit, and perhaps they are
>>>>>>>>>> not pound for
>>>>>>>>>> > pound the same.  For example, this test used to really be
>>>>>>>>>> marked XFAIL (via
>>>>>>>>>> > an expectedFailureClang directive), but it looks like the
>>>>>>>>>> current marking of
>>>>>>>>>> > compiler="clang" is either not right or not working, since the
>>>>>>>>>> test is run
>>>>>>>>>> > on OS X and is treated like it is expected to pass.
>>>>>>>>>> >
>>>>>>>>>> > I'm drilling into that a bit more, that's just the first of
>>>>>>>>>> several that
>>>>>>>>>> > fail with these changes on OS X.
>>>>>>>>>> >
>>>>>>>>>> > On Mon, Dec 14, 2015 at 3:03 PM, Zachary Turner <
>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> I've checked in r255567 which fixes a problem pointed out by
>>>>>>>>>> Siva.  It
>>>>>>>>>> >> doesn't sound related to in 255542, but looking at those logs
>>>>>>>>>> I can't really
>>>>>>>>>> >> tell how my CL would be related.  If r255567 doesn't fix the
>>>>>>>>>> bots, would
>>>>>>>>>> >> someone mind helping me briefly?  r255542 seems pretty
>>>>>>>>>> straightforward, so I
>>>>>>>>>> >> don't see why it would have an effect here.
>>>>>>>>>> >>
>>>>>>>>>> >> On Mon, Dec 14, 2015 at 2:35 PM Todd Fiala <
>>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> Ah yes I see.  Thanks, Ying (and Siva!  Saw your comments
>>>>>>>>>> too).
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Mon, Dec 14, 2015 at 2:34 PM, Ying Chen <chying at google.com>
>>>>>>>>>> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Seems this is the first build that fails, and it only has
>>>>>>>>>> one CL 255542.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446
>>>>>>>>>> >>>> I believe Zachary is looking at that problem.
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Mon, Dec 14, 2015 at 2:18 PM, Todd Fiala <
>>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>>> >>>> wrote:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> I am seeing several failures on the Ubuntu 14.04 testbot,
>>>>>>>>>> but
>>>>>>>>>> >>>>> unfortunately there are a number of changes that went in at
>>>>>>>>>> the same time on
>>>>>>>>>> >>>>> that build.  The failures I'm seeing are not appearing at
>>>>>>>>>> all related to the
>>>>>>>>>> >>>>> test running infrastructure.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Anybody with a fast Linux system able to take a look to see
>>>>>>>>>> what
>>>>>>>>>> >>>>> exactly is failing there?
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> -Todd
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> On Mon, Dec 14, 2015 at 1:39 PM, Todd Fiala <
>>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>>> >>>>> wrote:
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> Hi all,
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> I just put in the single-worker, low-load, follow-up test
>>>>>>>>>> run pass in
>>>>>>>>>> >>>>>> r255543.  Most of the work for it went in late last week,
>>>>>>>>>> this just mostly
>>>>>>>>>> >>>>>> flips it on.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> The feature works like this:
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> * First test phase works as before: run all tests using
>>>>>>>>>> whatever level
>>>>>>>>>> >>>>>> of concurrency is normally used.  (e.g. 8 works on an
>>>>>>>>>> 8-logical-core box).
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> * Any timeouts, failures, errors, or anything else that
>>>>>>>>>> would have
>>>>>>>>>> >>>>>> caused a test failure is eligible for rerun if either (1)
>>>>>>>>>> it was marked as a
>>>>>>>>>> >>>>>> flakey test via the flakey decorator, or (2) if the
>>>>>>>>>> --rerun-all-issues
>>>>>>>>>> >>>>>> command line flag is provided.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> * After the first test phase, if there are any tests that
>>>>>>>>>> met rerun
>>>>>>>>>> >>>>>> eligibility that would have caused a test failure, those
>>>>>>>>>> get run using a
>>>>>>>>>> >>>>>> serial test phase.  Their results will overwrite (i.e.
>>>>>>>>>> replace) the previous
>>>>>>>>>> >>>>>> result for the given test method.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> The net result should be that tests that were load
>>>>>>>>>> sensitive and
>>>>>>>>>> >>>>>> intermittently fail during the first higher-concurrency
>>>>>>>>>> test phase should
>>>>>>>>>> >>>>>> (in theory) pass in the second, single worker test phase
>>>>>>>>>> when the test suite
>>>>>>>>>> >>>>>> is only using a single worker.  This should make the test
>>>>>>>>>> suite generate
>>>>>>>>>> >>>>>> fewer false positives on test failure notification, which
>>>>>>>>>> should make
>>>>>>>>>> >>>>>> continuous integration servers (testbots) much more useful
>>>>>>>>>> in terms of
>>>>>>>>>> >>>>>> generating actionable signals caused by version control
>>>>>>>>>> changes to the lldb
>>>>>>>>>> >>>>>> or related sources.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> Please let me know if you see any issues with this when
>>>>>>>>>> running the
>>>>>>>>>> >>>>>> test suite using the default output.  I'd like to fix this
>>>>>>>>>> up ASAP.  And for
>>>>>>>>>> >>>>>> those interested in the implementation, I'm happy to do
>>>>>>>>>> post-commit
>>>>>>>>>> >>>>>> review/changes as needed to get it in good shape.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> I'll be watching the  builders now and will address any
>>>>>>>>>> issues as I
>>>>>>>>>> >>>>>> see them.
>>>>>>>>>> >>>>>>
>>>>>>>>>> >>>>>> Thanks!
>>>>>>>>>> >>>>>> --
>>>>>>>>>> >>>>>> -Todd
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> --
>>>>>>>>>> >>>>> -Todd
>>>>>>>>>> >>>>
>>>>>>>>>> >>>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>> --
>>>>>>>>>> >>> -Todd
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > -Todd
>>>>>>>>>> >
>>>>>>>>>> > _______________________________________________
>>>>>>>>>> > lldb-dev mailing list
>>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>> >
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>> _______________________________________________
>>>>>> lldb-dev mailing list
>>>>>> lldb-dev at lists.llvm.org
>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>
>
> --
> -Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151215/e0407801/attachment-0001.html>

More information about the lldb-dev mailing list