[lldb-dev] [Bug 24869] New: dotest.py reports the wrong number of test cases in mutiprocess mode.

[via lldb-dev]

https://llvm.org/bugs/show_bug.cgi?id=24869

            Bug ID: 24869
           Summary: dotest.py reports the wrong number of test cases in
                    mutiprocess mode.
           Product: lldb
           Version: unspecified
          Hardware: PC
                OS: All
            Status: NEW
          Severity: normal
          Priority: P
         Component: All Bugs
          Assignee: lldb-dev at lists.llvm.org
          Reporter: dawn+bugzilla at burble.org
                CC: llvm-bugs at lists.llvm.org
    Classification: Unclassified

This appears to result from dotest.py failing to count the test cases from
entire test files, as described in a run on a narrowed example below (following
the problem description).

** problem description and steps **

Run "dotest.py -v --output-on-success" and count the total test cases via
adding
up all the Ns in the lines:
        Ran N tests in .*
and you get the correct total, 1388 on OSX.
Compare that to the final count of 838 from:
        Ran N test cases .*
and you see a difference of 550 tests.

Commands used to run the tests and collect the results are as follows:
    cd llvm/tools/lldb/test
    ./dotest.py -v --output-on-success 2>&1 | tee $INSTALLDIR/lldb_test_out.log
    export lldb_failures=`grep -E "^RESULT:" lldb_test_out.log | grep failures
| awk '{count+=$5} END {print count}'` || true
    export lldb_errors=`grep -E "^RESULT:" lldb_test_out.log | grep errors |
awk '{count+=$7} END {print count}'` || true
    export lldb_total=`grep -E "^Ran [0-9]+ tests? in" lldb_test_out.log | awk
'{count+=$2} END {print count}'`
    export lldb_total_cases=`grep -E "^Ran [0-9]+ test cases" lldb_test_out.log
| awk '{count+=$2} END {print count}'`

On OSX, I get:
    lldb_failures=4
    lldb_errors=0
    lldb_total=1388
    lldb_total_cases=838


** narrowed example **

When run on the following narrowed subset of tests, you can see that dotest.py
is not counting any test cases from an entire test file.

Test on just 4 types test suites:
        rm -rf api arm_emulation attic benchmarks driver
        rm -rf example expression_command functionalities help lang
        rm -rf linux logging macosx python_api tools
        rm -rf settings source-manager terminal warnings
        rm -f types/TestIntegerTypes.py

You should have only the following tests left:
        types/AbstractBase.py
        types/HideTestFailures.py
        types/TestFloatTypes.py
        types/TestFloatTypesExpr.py
        types/TestIntegerTypesExpr.py
        types/TestRecursiveTypes.py

When dotest.py is run with the --no-multiprocess option, it correctly counts
46, but counts only 16 otherwise, as can been seen in the relevant test output:

    ./dotest.py -v --output-on-success --no-multiprocess:
        Ran 46 tests in 76.124s

    ./dotest.py -v --output-on-success:
        Testing: 4 test suites, 4 threads
        Ran 6 tests in 7.090s
        [TestFloatTypes.py PASSED]
        Ran 6 tests in 11.230s
        [TestFloatTypesExpr.py PASSED]
        Ran 30 tests in 55.200s
        [TestIntegerTypesExpr.py PASSED]
        Ran 4 tests in 3.608s
        [TestRecursiveTypes.py PASSED]
        Ran 4 test suites (0 failed) (0.000000%)
        Ran 16 test cases (0 failed) (0.000000%)

All 30 test cases from TestIntegerTypesExpr.py appear to have been omitted from
the total.

With run on only 3 test suites, both correctly count 16 test cases:

    rm -f types/TestIntegerTypesExpr.py
    ./dotest.py -v --output-on-success --no-multiprocess:
        Ran 16 tests in 21.426s
    ./dotest.py -v --output-on-success:
        Testing: 3 test suites, 4 threads
        Ran 6 tests in 7.245s
        [TestFloatTypes.py PASSED]
        Ran 6 tests in 11.474s
        [TestFloatTypesExpr.py PASSED]
        Ran 4 tests in 3.599s
        [TestRecursiveTypes.py PASSED]
        3 out of 3 test suites processed - TestRecursiveTypes.py
        Ran 3 test suites (0 failed) (0.000000%)
        Ran 16 test cases (0 failed) (0.000000%)

-- 
You are receiving this mail because:
You are the assignee for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150917/c50037f8/attachment-0001.html>