[lldb-dev] test rerun phase is in
Todd Fiala via lldb-dev
lldb-dev at lists.llvm.org
Mon Dec 14 16:52:50 PST 2015
And, btw, this shows the rerun logic working (via the --rerun-all-issues
flag):
time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24
--rerun-all-issues
Testing: 416 test suites, 24 threads
377 out of 416 test suites processed - TestSBTypeTypeClass.py
Session logs for test failures/errors/unexpected successes will go into
directory '2015-12-14-16_44_28'
Command invoked: test/dotest.py --executable
/Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
--rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
-p TestMultithreaded.py
/Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
--event-add-entries worker_index=3:int
Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 8 tests
lldb_codesign: no identity found
lldb_codesign: no identity found
lldb_codesign: no identity found
lldb_codesign: no identity found
lldb_codesign: no identity found
lldb_codesign: no identity found
lldb_codesign: no identity found
[TestMultithreaded.py FAILED]
Command invoked: /usr/bin/python test/dotest.py --executable
/Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
--rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
-p TestMultithreaded.py
/Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
--event-add-entries worker_index=3:int
396 out of 416 test suites processed - TestMiBreak.py
Session logs for test failures/errors/unexpected successes will go into
directory '2015-12-14-16_44_28'
Command invoked: test/dotest.py --executable
/Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
--rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
-p TestDataFormatterObjC.py
/Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
--event-add-entries worker_index=12:int
Configuration: arch=x86_64 compiler=clang
----------------------------------------------------------------------
Collected 26 tests
[TestDataFormatterObjC.py FAILED]
Command invoked: /usr/bin/python test/dotest.py --executable
/Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24
--rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior
-p TestDataFormatterObjC.py
/Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test
--event-add-entries worker_index=12:int
416 out of 416 test suites processed - TestLldbGdbServer.py
2 test files marked for rerun
Rerunning the following files:
functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py
api/multithreaded/TestMultithreaded.py
Testing: 2 test suites, 1 thread
2 out of 2 test suites processed - TestMultithreaded.py
Test rerun complete
=============
Issue Details
=============
UNEXPECTED SUCCESS: test_symbol_name_dsym
(functionalities/completion/TestCompletion.py)
UNEXPECTED SUCCESS: test_symbol_name_dwarf
(functionalities/completion/TestCompletion.py)
===================
Test Result Summary
===================
Test Methods: 1695
Reruns: 30
Success: 1367
Expected Failure: 90
Failure: 0
Error: 0
Exceptional Exit: 0
Unexpected Success: 2
Skip: 236
Timeout: 0
Expected Timeout: 0
On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
> And that fixed the rest as well. Thanks, Siva!
>
> -Todd
>
> On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> Heh you were skinning the same cat :-)
>>
>> That fixed the one I was just looking at, running the others now.
>>
>> On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala <todd.fiala at gmail.com> wrote:
>>
>>> Yep, will try now... (I was just looking at the condition testing logic
>>> since it looks like something isn't quite right there).
>>>
>>> On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra <sivachandra at google.com>
>>> wrote:
>>>
>>>> Can you try again after taking my change at r255584?
>>>>
>>>> On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev
>>>> <lldb-dev at lists.llvm.org> wrote:
>>>> > I'm having some of these blow up.
>>>> >
>>>> > In the case of test/lang/c/typedef/Testtypedef.py, it looks like some
>>>> of the
>>>> > @expected decorators were changed a bit, and perhaps they are not
>>>> pound for
>>>> > pound the same. For example, this test used to really be marked
>>>> XFAIL (via
>>>> > an expectedFailureClang directive), but it looks like the current
>>>> marking of
>>>> > compiler="clang" is either not right or not working, since the test
>>>> is run
>>>> > on OS X and is treated like it is expected to pass.
>>>> >
>>>> > I'm drilling into that a bit more, that's just the first of several
>>>> that
>>>> > fail with these changes on OS X.
>>>> >
>>>> > On Mon, Dec 14, 2015 at 3:03 PM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>> >>
>>>> >> I've checked in r255567 which fixes a problem pointed out by Siva.
>>>> It
>>>> >> doesn't sound related to in 255542, but looking at those logs I
>>>> can't really
>>>> >> tell how my CL would be related. If r255567 doesn't fix the bots,
>>>> would
>>>> >> someone mind helping me briefly? r255542 seems pretty
>>>> straightforward, so I
>>>> >> don't see why it would have an effect here.
>>>> >>
>>>> >> On Mon, Dec 14, 2015 at 2:35 PM Todd Fiala <todd.fiala at gmail.com>
>>>> wrote:
>>>> >>>
>>>> >>> Ah yes I see. Thanks, Ying (and Siva! Saw your comments too).
>>>> >>>
>>>> >>> On Mon, Dec 14, 2015 at 2:34 PM, Ying Chen <chying at google.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> Seems this is the first build that fails, and it only has one CL
>>>> 255542.
>>>> >>>>
>>>> >>>>
>>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446
>>>> >>>> I believe Zachary is looking at that problem.
>>>> >>>>
>>>> >>>> On Mon, Dec 14, 2015 at 2:18 PM, Todd Fiala <todd.fiala at gmail.com>
>>>> >>>> wrote:
>>>> >>>>>
>>>> >>>>> I am seeing several failures on the Ubuntu 14.04 testbot, but
>>>> >>>>> unfortunately there are a number of changes that went in at the
>>>> same time on
>>>> >>>>> that build. The failures I'm seeing are not appearing at all
>>>> related to the
>>>> >>>>> test running infrastructure.
>>>> >>>>>
>>>> >>>>> Anybody with a fast Linux system able to take a look to see what
>>>> >>>>> exactly is failing there?
>>>> >>>>>
>>>> >>>>> -Todd
>>>> >>>>>
>>>> >>>>> On Mon, Dec 14, 2015 at 1:39 PM, Todd Fiala <todd.fiala at gmail.com
>>>> >
>>>> >>>>> wrote:
>>>> >>>>>>
>>>> >>>>>> Hi all,
>>>> >>>>>>
>>>> >>>>>> I just put in the single-worker, low-load, follow-up test run
>>>> pass in
>>>> >>>>>> r255543. Most of the work for it went in late last week, this
>>>> just mostly
>>>> >>>>>> flips it on.
>>>> >>>>>>
>>>> >>>>>> The feature works like this:
>>>> >>>>>>
>>>> >>>>>> * First test phase works as before: run all tests using whatever
>>>> level
>>>> >>>>>> of concurrency is normally used. (e.g. 8 works on an
>>>> 8-logical-core box).
>>>> >>>>>>
>>>> >>>>>> * Any timeouts, failures, errors, or anything else that would
>>>> have
>>>> >>>>>> caused a test failure is eligible for rerun if either (1) it was
>>>> marked as a
>>>> >>>>>> flakey test via the flakey decorator, or (2) if the
>>>> --rerun-all-issues
>>>> >>>>>> command line flag is provided.
>>>> >>>>>>
>>>> >>>>>> * After the first test phase, if there are any tests that met
>>>> rerun
>>>> >>>>>> eligibility that would have caused a test failure, those get run
>>>> using a
>>>> >>>>>> serial test phase. Their results will overwrite (i.e. replace)
>>>> the previous
>>>> >>>>>> result for the given test method.
>>>> >>>>>>
>>>> >>>>>> The net result should be that tests that were load sensitive and
>>>> >>>>>> intermittently fail during the first higher-concurrency test
>>>> phase should
>>>> >>>>>> (in theory) pass in the second, single worker test phase when
>>>> the test suite
>>>> >>>>>> is only using a single worker. This should make the test suite
>>>> generate
>>>> >>>>>> fewer false positives on test failure notification, which should
>>>> make
>>>> >>>>>> continuous integration servers (testbots) much more useful in
>>>> terms of
>>>> >>>>>> generating actionable signals caused by version control changes
>>>> to the lldb
>>>> >>>>>> or related sources.
>>>> >>>>>>
>>>> >>>>>> Please let me know if you see any issues with this when running
>>>> the
>>>> >>>>>> test suite using the default output. I'd like to fix this up
>>>> ASAP. And for
>>>> >>>>>> those interested in the implementation, I'm happy to do
>>>> post-commit
>>>> >>>>>> review/changes as needed to get it in good shape.
>>>> >>>>>>
>>>> >>>>>> I'll be watching the builders now and will address any issues
>>>> as I
>>>> >>>>>> see them.
>>>> >>>>>>
>>>> >>>>>> Thanks!
>>>> >>>>>> --
>>>> >>>>>> -Todd
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> --
>>>> >>>>> -Todd
>>>> >>>>
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> -Todd
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > -Todd
>>>> >
>>>> > _______________________________________________
>>>> > lldb-dev mailing list
>>>> > lldb-dev at lists.llvm.org
>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>
--
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151214/47369746/attachment-0001.html>
More information about the lldb-dev
mailing list