[lldb-dev] test results look typical?

[Todd Fiala via lldb-dev]

Thanks for the details on dosep.py, Dawn.

For counting I will probably go back to my old method of parsing the output
of a serial dotest run, since IIRC I can get skip counts accurately there
as well.  (Or perhaps that should be added to dosep.py, it's been a while
since I last heavily modified that script).

-Todd

On Mon, Aug 24, 2015 at 6:50 PM, <dawn at burble.org> wrote:

> On Mon, Aug 24, 2015 at 05:37:43PM -0700, via lldb-dev wrote:
> > On Mon, Aug 24, 2015 at 03:37:52PM -0700, Todd Fiala via lldb-dev wrote:
> > > On Linux on non-virtualized hardware, I currently see the failures
> below on
> > > Ubuntu 14.04.2 using a setup like this:
> > > [...]
> > >
> > > ninja check-lldb output:
>
> FYI, ninja check-lldb actually calls dosep.
>
> > > Ran 394 test suites (15 failed) (3.807107%)
> > > Ran 474 test cases (17 failed) (3.586498%)
> >
> > I don't think you can trust the reporting of dosep.py's "Ran N test
> > cases", as it fails to count about 500 test cases.  The only way I've
> > found to get an accurate count is to add up all the Ns from "Ran N tests
> > in" as follows:
> >
> > ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee
> test_out.log
> > export total=`grep -E "^Ran [0-9]+ tests? in" test_out.log | awk
> '{count+=$2} END {print count}'`
>
> Of course, these commands assume you're running the tests from the
> lldb/test directory.
>
> > (See comments in http://reviews.llvm.org/rL238467.)
>
> I've pasted (and tweaked) the relavent comments from that review here,
> where I describe a narrowed case showing how dosep fails to count all the
> test cases from one test suite in test/types.  Note that the tests were run
> on OSX, so your counts may vary.
>
> The final count from:
>     Ran N test cases .*
> is wrong, as I'll explain below. I've done a comparison between dosep and
> dotest on a narrowed subset of tests to show how dosep can omit the test
> cases from a test suite in its count.
>
> Tested on subset of lldb/test with just the following directories/files
> (i.e. all others directories/files were removed):
>     test/make
>     test/pexpect-2.4
>     test/plugins
>     test/types
>     test/unittest2
> # The .py files kept in test/types are as follows (so
> test/types/TestIntegerTypes.py* was removed):
>     test/types/AbstractBase.py
>     test/types/HideTestFailures.py
>     test/types/TestFloatTypes.py
>     test/types/TestFloatTypesExpr.py
>     test/types/TestIntegerTypesExpr.py
>     test/types/TestRecursiveTypes.py
>
> Tests were run in the lldb/test directory using the following commands:
>     dotest:
>         ./dotest.py -v
>     dosep:
>         ./dosep.py -s --options "-v"
>
> Comparing the test case totals, dotest correctly counts 46, but dosep
> counts only 16:
>     dotest:
>         Ran 46 tests in 75.934s
>     dosep:
>         Testing: 23 tests, 4 threads ## note: this number changes randonmly
>         Ran 6 tests in 7.049s
>         [PASSED TestFloatTypes.py] - 1 out of 23 test suites processed
>         Ran 6 tests in 11.165s
>         [PASSED TestFloatTypesExpr.py] - 2 out of 23 test suites processed
>         Ran 30 tests in 54.581s ## FIXME: not counted?
>         [PASSED TestIntegerTypesExpr.py] - 3 out of 23 test suites
> processed
>         Ran 4 tests in 3.212s
>         [PASSED TestRecursiveTypes.py] - 4 out of 23 test suites processed
>         Ran 4 test suites (0 failed) (0.000000%)
>         Ran 16 test cases (0 failed) (0.000000%)
>
> With test/types/TestIntegerTypesExpr.py* removed, both correctly count 16
> test cases:
>     dosep:
>         Testing: 16 tests, 4 threads
>         Ran 6 tests in 7.059s
>         Ran 6 tests in 11.186s
>         Ran 4 tests in 3.241s
>         Ran 3 test suites (0 failed) (0.000000%)
>         Ran 16 test cases (0 failed) (0.000000%)
>
> Note: I couldn't compare the test counts on all the tests because of the
> concern raised in http://reviews.llvm.org/rL237053. That is, dotest can
> no longer complete the tests on OSX, as all test suites fail after test
> case 898: test_disassemble_invalid_vst_1_64_raw_data get ERRORs. I don't
> think that issue is related to problems in dosep.
>
> Thanks,
> -Dawn
>



-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150825/53aa1694/attachment.html>