[lldb-dev] test results look typical?

Tue Aug 25 09:41:34 PDT 2015

It would be great (and not too difficult) to add skip counts to dosep.  I
modified dotest so it formats the result summary in a nice single string
that you can regex match to get counts.  It's already matched in dosep, but
we just aren't pulling out the skip counts.  So it would be very easy to
add this.

On Tue, Aug 25, 2015 at 7:41 AM Todd Fiala via lldb-dev <
lldb-dev at lists.llvm.org> wrote:

> Thanks for the details on dosep.py, Dawn.
>
> For counting I will probably go back to my old method of parsing the
> output of a serial dotest run, since IIRC I can get skip counts accurately
> there as well.  (Or perhaps that should be added to dosep.py, it's been a
> while since I last heavily modified that script).
>
> -Todd
>
> On Mon, Aug 24, 2015 at 6:50 PM, <dawn at burble.org> wrote:
>
>> On Mon, Aug 24, 2015 at 05:37:43PM -0700, via lldb-dev wrote:
>> > On Mon, Aug 24, 2015 at 03:37:52PM -0700, Todd Fiala via lldb-dev wrote:
>> > > On Linux on non-virtualized hardware, I currently see the failures
>> below on
>> > > Ubuntu 14.04.2 using a setup like this:
>> > > [...]
>> > >
>> > > ninja check-lldb output:
>>
>> FYI, ninja check-lldb actually calls dosep.
>>
>> > > Ran 394 test suites (15 failed) (3.807107%)
>> > > Ran 474 test cases (17 failed) (3.586498%)
>> >
>> > I don't think you can trust the reporting of dosep.py's "Ran N test
>> > cases", as it fails to count about 500 test cases.  The only way I've
>> > found to get an accurate count is to add up all the Ns from "Ran N tests
>> > in" as follows:
>> >
>> > ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee
>> test_out.log
>> > export total=`grep -E "^Ran [0-9]+ tests? in" test_out.log | awk
>> '{count+=$2} END {print count}'`
>>
>> Of course, these commands assume you're running the tests from the
>> lldb/test directory.
>>
>> > (See comments in http://reviews.llvm.org/rL238467.)
>>
>> I've pasted (and tweaked) the relavent comments from that review here,
>> where I describe a narrowed case showing how dosep fails to count all the
>> test cases from one test suite in test/types.  Note that the tests were run
>> on OSX, so your counts may vary.
>>
>> The final count from:
>>     Ran N test cases .*
>> is wrong, as I'll explain below. I've done a comparison between dosep and
>> dotest on a narrowed subset of tests to show how dosep can omit the test
>> cases from a test suite in its count.
>>
>> Tested on subset of lldb/test with just the following directories/files
>> (i.e. all others directories/files were removed):
>>     test/make
>>     test/pexpect-2.4
>>     test/plugins
>>     test/types
>>     test/unittest2
>> # The .py files kept in test/types are as follows (so
>> test/types/TestIntegerTypes.py* was removed):
>>     test/types/AbstractBase.py
>>     test/types/HideTestFailures.py
>>     test/types/TestFloatTypes.py
>>     test/types/TestFloatTypesExpr.py
>>     test/types/TestIntegerTypesExpr.py
>>     test/types/TestRecursiveTypes.py
>>
>> Tests were run in the lldb/test directory using the following commands:
>>     dotest:
>>         ./dotest.py -v
>>     dosep:
>>         ./dosep.py -s --options "-v"
>>
>> Comparing the test case totals, dotest correctly counts 46, but dosep
>> counts only 16:
>>     dotest:
>>         Ran 46 tests in 75.934s
>>     dosep:
>>         Testing: 23 tests, 4 threads ## note: this number changes
>> randonmly
>>         Ran 6 tests in 7.049s
>>         [PASSED TestFloatTypes.py] - 1 out of 23 test suites processed
>>         Ran 6 tests in 11.165s
>>         [PASSED TestFloatTypesExpr.py] - 2 out of 23 test suites processed
>>         Ran 30 tests in 54.581s ## FIXME: not counted?
>>         [PASSED TestIntegerTypesExpr.py] - 3 out of 23 test suites
>> processed
>>         Ran 4 tests in 3.212s
>>         [PASSED TestRecursiveTypes.py] - 4 out of 23 test suites processed
>>         Ran 4 test suites (0 failed) (0.000000%)
>>         Ran 16 test cases (0 failed) (0.000000%)
>>
>> With test/types/TestIntegerTypesExpr.py* removed, both correctly count 16
>> test cases:
>>     dosep:
>>         Testing: 16 tests, 4 threads
>>         Ran 6 tests in 7.059s
>>         Ran 6 tests in 11.186s
>>         Ran 4 tests in 3.241s
>>         Ran 3 test suites (0 failed) (0.000000%)
>>         Ran 16 test cases (0 failed) (0.000000%)
>>
>> Note: I couldn't compare the test counts on all the tests because of the
>> concern raised in http://reviews.llvm.org/rL237053. That is, dotest can
>> no longer complete the tests on OSX, as all test suites fail after test
>> case 898: test_disassemble_invalid_vst_1_64_raw_data get ERRORs. I don't
>> think that issue is related to problems in dosep.
>>
>> Thanks,
>> -Dawn
>>
>
>
>
> --
> -Todd
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150825/ae08d048/attachment.html>