<div dir="ltr">Thanks for the details on dosep.py, Dawn.<div><br></div><div>For counting I will probably go back to my old method of parsing the output of a serial dotest run, since IIRC I can get skip counts accurately there as well. (Or perhaps that should be added to dosep.py, it's been a while since I last heavily modified that script).</div><div><br></div><div>-Todd</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 24, 2015 at 6:50 PM, <span dir="ltr"><<a href="mailto:dawn@burble.org" target="_blank">dawn@burble.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Mon, Aug 24, 2015 at 05:37:43PM -0700, via lldb-dev wrote:<br>
> On Mon, Aug 24, 2015 at 03:37:52PM -0700, Todd Fiala via lldb-dev wrote:<br>
> > On Linux on non-virtualized hardware, I currently see the failures below on<br>
> > Ubuntu 14.04.2 using a setup like this:<br>
> > [...]<br>
> ><br>
> > ninja check-lldb output:<br>
<br>
</span>FYI, ninja check-lldb actually calls dosep.<br>
<span class=""><br>
> > Ran 394 test suites (15 failed) (3.807107%)<br>
> > Ran 474 test cases (17 failed) (3.586498%)<br>
><br>
</span>> I don't think you can trust the reporting of dosep.py's "Ran N test<br>
<span class="">> cases", as it fails to count about 500 test cases. The only way I've<br>
> found to get an accurate count is to add up all the Ns from "Ran N tests<br>
> in" as follows:<br>
><br>
> ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee test_out.log<br>
> export total=`grep -E "^Ran [0-9]+ tests? in" test_out.log | awk '{count+=$2} END {print count}'`<br>
<br>
</span>Of course, these commands assume you're running the tests from the lldb/test directory.<br>
<span class=""><br>
> (See comments in <a href="http://reviews.llvm.org/rL238467" rel="noreferrer" target="_blank">http://reviews.llvm.org/rL238467</a>.)<br>
<br>
</span>I've pasted (and tweaked) the relavent comments from that review here, where I describe a narrowed case showing how dosep fails to count all the test cases from one test suite in test/types. Note that the tests were run on OSX, so your counts may vary.<br>
<br>
The final count from:<br>
Ran N test cases .*<br>
is wrong, as I'll explain below. I've done a comparison between dosep and dotest on a narrowed subset of tests to show how dosep can omit the test cases from a test suite in its count.<br>
<br>
Tested on subset of lldb/test with just the following directories/files (i.e. all others directories/files were removed):<br>
test/make<br>
test/pexpect-2.4<br>
test/plugins<br>
test/types<br>
test/unittest2<br>
# The .py files kept in test/types are as follows (so test/types/TestIntegerTypes.py* was removed):<br>
test/types/AbstractBase.py<br>
test/types/HideTestFailures.py<br>
test/types/TestFloatTypes.py<br>
test/types/TestFloatTypesExpr.py<br>
test/types/TestIntegerTypesExpr.py<br>
test/types/TestRecursiveTypes.py<br>
<br>
Tests were run in the lldb/test directory using the following commands:<br>
dotest:<br>
./dotest.py -v<br>
dosep:<br>
./dosep.py -s --options "-v"<br>
<br>
Comparing the test case totals, dotest correctly counts 46, but dosep counts only 16:<br>
dotest:<br>
Ran 46 tests in 75.934s<br>
dosep:<br>
Testing: 23 tests, 4 threads ## note: this number changes randonmly<br>
Ran 6 tests in 7.049s<br>
[PASSED TestFloatTypes.py] - 1 out of 23 test suites processed<br>
Ran 6 tests in 11.165s<br>
[PASSED TestFloatTypesExpr.py] - 2 out of 23 test suites processed<br>
Ran 30 tests in 54.581s ## FIXME: not counted?<br>
[PASSED TestIntegerTypesExpr.py] - 3 out of 23 test suites processed<br>
Ran 4 tests in 3.212s<br>
[PASSED TestRecursiveTypes.py] - 4 out of 23 test suites processed<br>
Ran 4 test suites (0 failed) (0.000000%)<br>
Ran 16 test cases (0 failed) (0.000000%)<br>
<br>
With test/types/TestIntegerTypesExpr.py* removed, both correctly count 16 test cases:<br>
dosep:<br>
Testing: 16 tests, 4 threads<br>
Ran 6 tests in 7.059s<br>
Ran 6 tests in 11.186s<br>
Ran 4 tests in 3.241s<br>
Ran 3 test suites (0 failed) (0.000000%)<br>
Ran 16 test cases (0 failed) (0.000000%)<br>
<br>
Note: I couldn't compare the test counts on all the tests because of the concern raised in <a href="http://reviews.llvm.org/rL237053" rel="noreferrer" target="_blank">http://reviews.llvm.org/rL237053</a>. That is, dotest can no longer complete the tests on OSX, as all test suites fail after test case 898: test_disassemble_invalid_vst_1_64_raw_data get ERRORs. I don't think that issue is related to problems in dosep.<br>
<br>
Thanks,<br>
-Dawn<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">-Todd</div></div>
</div>