[lldb-dev] test rerun phase is in

Todd Fiala via lldb-dev lldb-dev at lists.llvm.org
Tue Dec 15 14:34:27 PST 2015 

Build >= #4310 is what I'll be watching.


On Tue, Dec 15, 2015 at 2:30 PM, Todd Fiala <todd.fiala at gmail.com> wrote:

> Okay cool.  Will do.
>
> On Tue, Dec 15, 2015 at 2:22 PM, Ying Chen <chying at google.com> wrote:
>
>> Sure. Please go ahead to do that.
>> BTW, the pending builds should be merged into one build once current
>> build is done.
>>
>> On Tue, Dec 15, 2015 at 2:12 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> Hey Ying,
>>>
>>> Do you mind if we clear the android builder queue to get a build with
>>> r255676 in it?  There are what looks like at least 3 or 4 builds between
>>> now and then, and with timeouts it may take several hours.
>>>
>>> -Todd
>>>
>>> On Tue, Dec 15, 2015 at 1:50 PM, Ying Chen <chying at google.com> wrote:
>>>
>>>> Yes, it happens every time for android builder.
>>>>
>>>> On Tue, Dec 15, 2015 at 1:45 PM, Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>>
>>>>> Hmm, yeah it looks like it did the rerun and then after finishing the
>>>>> rerun, it's just hanging.
>>>>>
>>>>> Let's have a look right after r255676 goes through this builder.  I
>>>>> hit a hang in the curses output display due to the recursive taking of a
>>>>> lock on a lock that was not recursive-enabled.  While I would have expected
>>>>> to see that with the basic results output that this builder here is using
>>>>> when I was testing earlier, it's possible somehow that we're hitting a path
>>>>> here that is attempting to recursively take a lock.
>>>>>
>>>>> Do you know if it is happening every single time a rerun occurs?
>>>>>  (Hopefully yes?)
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Yep, I'll have a look!
>>>>>>
>>>>>> On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen <chying at google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Todd,
>>>>>>>
>>>>>>> It is noticed on lldb android builders that the test_runner didn't
>>>>>>> exit after rerun, which caused buildbot timeout since the process was
>>>>>>> hanging for over 20 minutes.
>>>>>>> Could you please take a look if that's related to your change?
>>>>>>>
>>>>>>> Please see the following builds.
>>>>>>>
>>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio
>>>>>>>
>>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Ying
>>>>>>>
>>>>>>> On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <
>>>>>>> lldb-dev at lists.llvm.org> wrote:
>>>>>>>
>>>>>>>> And, btw, this shows the rerun logic working (via the
>>>>>>>> --rerun-all-issues flag):
>>>>>>>>
>>>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb --threads
>>>>>>>> 24 --rerun-all-issues
>>>>>>>> Testing: 416 test suites, 24 threads
>>>>>>>> 377 out of 416 test suites processed - TestSBTypeTypeClass.py
>>>>>>>>
>>>>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>>>>> into directory '2015-12-14-16_44_28'
>>>>>>>> Command invoked: test/dotest.py --executable
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>>> -p TestMultithreaded.py
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>>> --event-add-entries worker_index=3:int
>>>>>>>>
>>>>>>>> Configuration: arch=x86_64 compiler=clang
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>> Collected 8 tests
>>>>>>>>
>>>>>>>> lldb_codesign: no identity found
>>>>>>>> lldb_codesign: no identity found
>>>>>>>> lldb_codesign: no identity found
>>>>>>>> lldb_codesign: no identity found
>>>>>>>> lldb_codesign: no identity found
>>>>>>>> lldb_codesign: no identity found
>>>>>>>> lldb_codesign: no identity found
>>>>>>>>
>>>>>>>> [TestMultithreaded.py FAILED]
>>>>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>>> -p TestMultithreaded.py
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>>> --event-add-entries worker_index=3:int
>>>>>>>> 396 out of 416 test suites processed - TestMiBreak.py
>>>>>>>>
>>>>>>>> Session logs for test failures/errors/unexpected successes will go
>>>>>>>> into directory '2015-12-14-16_44_28'
>>>>>>>> Command invoked: test/dotest.py --executable
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>>> -p TestDataFormatterObjC.py
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>>> --event-add-entries worker_index=12:int
>>>>>>>>
>>>>>>>> Configuration: arch=x86_64 compiler=clang
>>>>>>>>
>>>>>>>> ----------------------------------------------------------------------
>>>>>>>> Collected 26 tests
>>>>>>>>
>>>>>>>>
>>>>>>>> [TestDataFormatterObjC.py FAILED]
>>>>>>>> Command invoked: /usr/bin/python test/dotest.py --executable
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
>>>>>>>> --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
>>>>>>>> -p TestDataFormatterObjC.py
>>>>>>>> /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
>>>>>>>> --event-add-entries worker_index=12:int
>>>>>>>> 416 out of 416 test suites processed - TestLldbGdbServer.py
>>>>>>>> 2 test files marked for rerun
>>>>>>>>
>>>>>>>>
>>>>>>>> Rerunning the following files:
>>>>>>>>
>>>>>>>> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
>>>>>>>>   api/multithreaded/TestMultithreaded.py
>>>>>>>> Testing: 2 test suites, 1 thread
>>>>>>>> 2 out of 2 test suites processed - TestMultithreaded.py
>>>>>>>> Test rerun complete
>>>>>>>>
>>>>>>>>
>>>>>>>> =============
>>>>>>>> Issue Details
>>>>>>>> =============
>>>>>>>> UNEXPECTED SUCCESS: test_symbol_name_dsym
>>>>>>>> (functionalities/completion/TestCompletion.py)
>>>>>>>> UNEXPECTED SUCCESS: test_symbol_name_dwarf
>>>>>>>> (functionalities/completion/TestCompletion.py)
>>>>>>>>
>>>>>>>> ===================
>>>>>>>> Test Result Summary
>>>>>>>> ===================
>>>>>>>> Test Methods:       1695
>>>>>>>> Reruns:               30
>>>>>>>> Success:            1367
>>>>>>>> Expected Failure:     90
>>>>>>>> Failure:               0
>>>>>>>> Error:                 0
>>>>>>>> Exceptional Exit:      0
>>>>>>>> Unexpected Success:    2
>>>>>>>> Skip:                236
>>>>>>>> Timeout:               0
>>>>>>>> Expected Timeout:      0
>>>>>>>>
>>>>>>>> On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> And that fixed the rest as well.  Thanks, Siva!
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Heh you were skinning the same cat :-)
>>>>>>>>>>
>>>>>>>>>> That fixed the one I was just looking at, running the others now.
>>>>>>>>>>
>>>>>>>>>> On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala <todd.fiala at gmail.com
>>>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Yep, will try now...  (I was just looking at the condition
>>>>>>>>>>> testing logic since it looks like something isn't quite right there).
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra <
>>>>>>>>>>> sivachandra at google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you try again after taking my change at r255584?
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev
>>>>>>>>>>>> <lldb-dev at lists.llvm.org> wrote:
>>>>>>>>>>>> > I'm having some of these blow up.
>>>>>>>>>>>> >
>>>>>>>>>>>> > In the case of test/lang/c/typedef/Testtypedef.py, it looks
>>>>>>>>>>>> like some of the
>>>>>>>>>>>> > @expected decorators were changed a bit, and perhaps they are
>>>>>>>>>>>> not pound for
>>>>>>>>>>>> > pound the same.  For example, this test used to really be
>>>>>>>>>>>> marked XFAIL (via
>>>>>>>>>>>> > an expectedFailureClang directive), but it looks like the
>>>>>>>>>>>> current marking of
>>>>>>>>>>>> > compiler="clang" is either not right or not working, since
>>>>>>>>>>>> the test is run
>>>>>>>>>>>> > on OS X and is treated like it is expected to pass.
>>>>>>>>>>>> >
>>>>>>>>>>>> > I'm drilling into that a bit more, that's just the first of
>>>>>>>>>>>> several that
>>>>>>>>>>>> > fail with these changes on OS X.
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Mon, Dec 14, 2015 at 3:03 PM, Zachary Turner <
>>>>>>>>>>>> zturner at google.com> wrote:
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> I've checked in r255567 which fixes a problem pointed out by
>>>>>>>>>>>> Siva.  It
>>>>>>>>>>>> >> doesn't sound related to in 255542, but looking at those
>>>>>>>>>>>> logs I can't really
>>>>>>>>>>>> >> tell how my CL would be related.  If r255567 doesn't fix the
>>>>>>>>>>>> bots, would
>>>>>>>>>>>> >> someone mind helping me briefly?  r255542 seems pretty
>>>>>>>>>>>> straightforward, so I
>>>>>>>>>>>> >> don't see why it would have an effect here.
>>>>>>>>>>>> >>
>>>>>>>>>>>> >> On Mon, Dec 14, 2015 at 2:35 PM Todd Fiala <
>>>>>>>>>>>> todd.fiala at gmail.com> wrote:
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> Ah yes I see.  Thanks, Ying (and Siva!  Saw your comments
>>>>>>>>>>>> too).
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> On Mon, Dec 14, 2015 at 2:34 PM, Ying Chen <
>>>>>>>>>>>> chying at google.com> wrote:
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> Seems this is the first build that fails, and it only has
>>>>>>>>>>>> one CL 255542.
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446
>>>>>>>>>>>> >>>> I believe Zachary is looking at that problem.
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>> On Mon, Dec 14, 2015 at 2:18 PM, Todd Fiala <
>>>>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>>>>> >>>> wrote:
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> I am seeing several failures on the Ubuntu 14.04 testbot,
>>>>>>>>>>>> but
>>>>>>>>>>>> >>>>> unfortunately there are a number of changes that went in
>>>>>>>>>>>> at the same time on
>>>>>>>>>>>> >>>>> that build.  The failures I'm seeing are not appearing at
>>>>>>>>>>>> all related to the
>>>>>>>>>>>> >>>>> test running infrastructure.
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> Anybody with a fast Linux system able to take a look to
>>>>>>>>>>>> see what
>>>>>>>>>>>> >>>>> exactly is failing there?
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> -Todd
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> On Mon, Dec 14, 2015 at 1:39 PM, Todd Fiala <
>>>>>>>>>>>> todd.fiala at gmail.com>
>>>>>>>>>>>> >>>>> wrote:
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> Hi all,
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> I just put in the single-worker, low-load, follow-up
>>>>>>>>>>>> test run pass in
>>>>>>>>>>>> >>>>>> r255543.  Most of the work for it went in late last
>>>>>>>>>>>> week, this just mostly
>>>>>>>>>>>> >>>>>> flips it on.
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> The feature works like this:
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> * First test phase works as before: run all tests using
>>>>>>>>>>>> whatever level
>>>>>>>>>>>> >>>>>> of concurrency is normally used.  (e.g. 8 works on an
>>>>>>>>>>>> 8-logical-core box).
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> * Any timeouts, failures, errors, or anything else that
>>>>>>>>>>>> would have
>>>>>>>>>>>> >>>>>> caused a test failure is eligible for rerun if either
>>>>>>>>>>>> (1) it was marked as a
>>>>>>>>>>>> >>>>>> flakey test via the flakey decorator, or (2) if the
>>>>>>>>>>>> --rerun-all-issues
>>>>>>>>>>>> >>>>>> command line flag is provided.
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> * After the first test phase, if there are any tests
>>>>>>>>>>>> that met rerun
>>>>>>>>>>>> >>>>>> eligibility that would have caused a test failure, those
>>>>>>>>>>>> get run using a
>>>>>>>>>>>> >>>>>> serial test phase.  Their results will overwrite (i.e.
>>>>>>>>>>>> replace) the previous
>>>>>>>>>>>> >>>>>> result for the given test method.
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> The net result should be that tests that were load
>>>>>>>>>>>> sensitive and
>>>>>>>>>>>> >>>>>> intermittently fail during the first higher-concurrency
>>>>>>>>>>>> test phase should
>>>>>>>>>>>> >>>>>> (in theory) pass in the second, single worker test phase
>>>>>>>>>>>> when the test suite
>>>>>>>>>>>> >>>>>> is only using a single worker.  This should make the
>>>>>>>>>>>> test suite generate
>>>>>>>>>>>> >>>>>> fewer false positives on test failure notification,
>>>>>>>>>>>> which should make
>>>>>>>>>>>> >>>>>> continuous integration servers (testbots) much more
>>>>>>>>>>>> useful in terms of
>>>>>>>>>>>> >>>>>> generating actionable signals caused by version control
>>>>>>>>>>>> changes to the lldb
>>>>>>>>>>>> >>>>>> or related sources.
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> Please let me know if you see any issues with this when
>>>>>>>>>>>> running the
>>>>>>>>>>>> >>>>>> test suite using the default output.  I'd like to fix
>>>>>>>>>>>> this up ASAP.  And for
>>>>>>>>>>>> >>>>>> those interested in the implementation, I'm happy to do
>>>>>>>>>>>> post-commit
>>>>>>>>>>>> >>>>>> review/changes as needed to get it in good shape.
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> I'll be watching the  builders now and will address any
>>>>>>>>>>>> issues as I
>>>>>>>>>>>> >>>>>> see them.
>>>>>>>>>>>> >>>>>>
>>>>>>>>>>>> >>>>>> Thanks!
>>>>>>>>>>>> >>>>>> --
>>>>>>>>>>>> >>>>>> -Todd
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>>
>>>>>>>>>>>> >>>>> --
>>>>>>>>>>>> >>>>> -Todd
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>>
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>>
>>>>>>>>>>>> >>> --
>>>>>>>>>>>> >>> -Todd
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > --
>>>>>>>>>>>> > -Todd
>>>>>>>>>>>> >
>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>> > lldb-dev mailing list
>>>>>>>>>>>> > lldb-dev at lists.llvm.org
>>>>>>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>>>> >
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> lldb-dev mailing list
>>>>>>>> lldb-dev at lists.llvm.org
>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>
>
> --
> -Todd
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151215/da124e16/attachment-0001.html>

More information about the lldb-dev mailing list