<html>
<head>
<base href="https://llvm.org/bugs/" />
</head>
<body><table border="1" cellspacing="0" cellpadding="8">
<tr>
<th>Bug ID</th>
<td><a class="bz_bug_link
bz_status_NEW "
title="NEW --- - dotest.py reports the wrong number of test cases in mutiprocess mode."
href="https://llvm.org/bugs/show_bug.cgi?id=24869">24869</a>
</td>
</tr>
<tr>
<th>Summary</th>
<td>dotest.py reports the wrong number of test cases in mutiprocess mode.
</td>
</tr>
<tr>
<th>Product</th>
<td>lldb
</td>
</tr>
<tr>
<th>Version</th>
<td>unspecified
</td>
</tr>
<tr>
<th>Hardware</th>
<td>PC
</td>
</tr>
<tr>
<th>OS</th>
<td>All
</td>
</tr>
<tr>
<th>Status</th>
<td>NEW
</td>
</tr>
<tr>
<th>Severity</th>
<td>normal
</td>
</tr>
<tr>
<th>Priority</th>
<td>P
</td>
</tr>
<tr>
<th>Component</th>
<td>All Bugs
</td>
</tr>
<tr>
<th>Assignee</th>
<td>lldb-dev@lists.llvm.org
</td>
</tr>
<tr>
<th>Reporter</th>
<td>dawn+bugzilla@burble.org
</td>
</tr>
<tr>
<th>CC</th>
<td>llvm-bugs@lists.llvm.org
</td>
</tr>
<tr>
<th>Classification</th>
<td>Unclassified
</td>
</tr></table>
<p>
<div>
<pre>This appears to result from dotest.py failing to count the test cases from
entire test files, as described in a run on a narrowed example below (following
the problem description).
** problem description and steps **
Run "dotest.py -v --output-on-success" and count the total test cases via
adding
up all the Ns in the lines:
Ran N tests in .*
and you get the correct total, 1388 on OSX.
Compare that to the final count of 838 from:
Ran N test cases .*
and you see a difference of 550 tests.
Commands used to run the tests and collect the results are as follows:
cd llvm/tools/lldb/test
./dotest.py -v --output-on-success 2>&1 | tee $INSTALLDIR/lldb_test_out.log
export lldb_failures=`grep -E "^RESULT:" lldb_test_out.log | grep failures
| awk '{count+=$5} END {print count}'` || true
export lldb_errors=`grep -E "^RESULT:" lldb_test_out.log | grep errors |
awk '{count+=$7} END {print count}'` || true
export lldb_total=`grep -E "^Ran [0-9]+ tests? in" lldb_test_out.log | awk
'{count+=$2} END {print count}'`
export lldb_total_cases=`grep -E "^Ran [0-9]+ test cases" lldb_test_out.log
| awk '{count+=$2} END {print count}'`
On OSX, I get:
lldb_failures=4
lldb_errors=0
lldb_total=1388
lldb_total_cases=838
** narrowed example **
When run on the following narrowed subset of tests, you can see that dotest.py
is not counting any test cases from an entire test file.
Test on just 4 types test suites:
rm -rf api arm_emulation attic benchmarks driver
rm -rf example expression_command functionalities help lang
rm -rf linux logging macosx python_api tools
rm -rf settings source-manager terminal warnings
rm -f types/TestIntegerTypes.py
You should have only the following tests left:
types/AbstractBase.py
types/HideTestFailures.py
types/TestFloatTypes.py
types/TestFloatTypesExpr.py
types/TestIntegerTypesExpr.py
types/TestRecursiveTypes.py
When dotest.py is run with the --no-multiprocess option, it correctly counts
46, but counts only 16 otherwise, as can been seen in the relevant test output:
./dotest.py -v --output-on-success --no-multiprocess:
Ran 46 tests in 76.124s
./dotest.py -v --output-on-success:
Testing: 4 test suites, 4 threads
Ran 6 tests in 7.090s
[TestFloatTypes.py PASSED]
Ran 6 tests in 11.230s
[TestFloatTypesExpr.py PASSED]
Ran 30 tests in 55.200s
[TestIntegerTypesExpr.py PASSED]
Ran 4 tests in 3.608s
[TestRecursiveTypes.py PASSED]
Ran 4 test suites (0 failed) (0.000000%)
Ran 16 test cases (0 failed) (0.000000%)
All 30 test cases from TestIntegerTypesExpr.py appear to have been omitted from
the total.
With run on only 3 test suites, both correctly count 16 test cases:
rm -f types/TestIntegerTypesExpr.py
./dotest.py -v --output-on-success --no-multiprocess:
Ran 16 tests in 21.426s
./dotest.py -v --output-on-success:
Testing: 3 test suites, 4 threads
Ran 6 tests in 7.245s
[TestFloatTypes.py PASSED]
Ran 6 tests in 11.474s
[TestFloatTypesExpr.py PASSED]
Ran 4 tests in 3.599s
[TestRecursiveTypes.py PASSED]
3 out of 3 test suites processed - TestRecursiveTypes.py
Ran 3 test suites (0 failed) (0.000000%)
Ran 16 test cases (0 failed) (0.000000%)</pre>
</div>
</p>
<hr>
<span>You are receiving this mail because:</span>
<ul>
<li>You are the assignee for the bug.</li>
</ul>
</body>
</html>