<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 25, 2015 at 5:40 AM, Tamas Berghammer <span dir="ltr"><<a href="mailto:tberghammer@google.com" target="_blank">tberghammer@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Going back to the original question I think you have more test failures then expected. As Chaoren mentioned all <span style="font-size:12.8000001907349px;line-height:19.2000007629395px">TestDataFormatterLibc* tests are failing because of a missing dependency,</span></div></blockquote><div><br></div><div>Thanks, Tamas. I'm going to be testing again today with libc++ installed.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span style="font-size:12.8000001907349px;line-height:19.2000007629395px"> but I think the rest of the tests should pass (I wouldn't expect them to depend on </span><span style="font-size:12.8000001907349px;line-height:19.2000007629395px">libc++-dev).</span><div><br></div></div></blockquote><div><br></div><div>I'll get a better handle on what's failing once I get rid of that first batch. </div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span style="font-size:12.8000001907349px;line-height:19.2000007629395px"></span></div><div><span style="font-size:12.8000001907349px;line-height:19.2000007629395px">You can see the up to date list of failures on the Linux buildbot here:</span></div><div><span style="font-size:12.8000001907349px;line-height:19.2000007629395px"><a href="http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake" target="_blank">http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake</a></span><br></div><div><br></div></div></blockquote><div><br></div><div>Ah yes,that'll be good to cross reference.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div>The buildbot is running in "Google Compute Engine" with Linux version: "Linux buildbot-master-ubuntu-1404 3.16.0-31-generic #43~14.04.1-Ubuntu SMP Tue Mar 10 20:13:38 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux"<div><br></div><div>LLDB is compiled by Clang (not sure about witch version but can find out if somebody thinks it matters) and the inferiors are compiled with clang-3.5, clang-tot, gcc-4.9.2. In all tested configuration there should be no failure (all failing tests should be XFAIL-ed).</div><div><br></div></div></blockquote><div><br></div><div>Ah okay good to know. In the past IIRC I did get different failures using clang-built vs. gcc-built lldb on Ubuntu 14.04. The clang-built lldbs at the time were harder to debug on Linux for one reason or another (I think particularly if any optimizations were enabled due to loss of debuginfo, but there might have been more). Are you using a clang-built lldb and debugging it reasonably well on Linux? If so I'd just assume move over to using clang so there's one less difference when I'm looking across platforms.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>For the flaky tests we introduced an "expectedFlaky" decorator what executes the test twice and expects it to pass at least once,</div></div></blockquote><div><br></div><div>Ah that's a good addition. We had talked about doing something to watch tests over time to see when it might be good to promote an XFAIL test that is consistently passing to a static "expect success" test. The flaky flag sounds handy for those that flap.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div> but it haven't been applied to all flaky test yet. The plan with the tests passing with "unexpected success" at the moment is to gather statistics about them and based on that mark them as "expected flaky" or remove the "expected failure" based on the number of failures we seen in the last few hundreds runs.</div></div></blockquote><div><br></div><div>Ah yes that :-) Love it.</div><div><br></div><div>Thanks, Tamas!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span class="HOEnZb"><font color="#888888"><div><br></div></font></span><div><span class="HOEnZb"><font color="#888888">Tamas<br></font></span><div><br><div class="gmail_quote"><div><div class="h5"><div dir="ltr">On Tue, Aug 25, 2015 at 2:50 AM via lldb-dev <<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">On Mon, Aug 24, 2015 at 05:37:43PM -0700, via lldb-dev wrote:<br>
> On Mon, Aug 24, 2015 at 03:37:52PM -0700, Todd Fiala via lldb-dev wrote:<br>
> > On Linux on non-virtualized hardware, I currently see the failures below on<br>
> > Ubuntu 14.04.2 using a setup like this:<br>
> > [...]<br>
> ><br>
> > ninja check-lldb output:<br>
<br>
FYI, ninja check-lldb actually calls dosep.<br>
<br>
> > Ran 394 test suites (15 failed) (3.807107%)<br>
> > Ran 474 test cases (17 failed) (3.586498%)<br>
><br>
> I don't think you can trust the reporting of dosep.py's "Ran N test<br>
> cases", as it fails to count about 500 test cases. The only way I've<br>
> found to get an accurate count is to add up all the Ns from "Ran N tests<br>
> in" as follows:<br>
><br>
> ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee test_out.log<br>
> export total=`grep -E "^Ran [0-9]+ tests? in" test_out.log | awk '{count+=$2} END {print count}'`<br>
<br>
Of course, these commands assume you're running the tests from the lldb/test directory.<br>
<br>
> (See comments in <a href="http://reviews.llvm.org/rL238467" rel="noreferrer" target="_blank">http://reviews.llvm.org/rL238467</a>.)<br>
<br>
I've pasted (and tweaked) the relavent comments from that review here, where I describe a narrowed case showing how dosep fails to count all the test cases from one test suite in test/types. Note that the tests were run on OSX, so your counts may vary.<br>
<br>
The final count from:<br>
Ran N test cases .*<br>
is wrong, as I'll explain below. I've done a comparison between dosep and dotest on a narrowed subset of tests to show how dosep can omit the test cases from a test suite in its count.<br>
<br>
Tested on subset of lldb/test with just the following directories/files (i.e. all others directories/files were removed):<br>
test/make<br>
test/pexpect-2.4<br>
test/plugins<br>
test/types<br>
test/unittest2<br>
# The .py files kept in test/types are as follows (so test/types/TestIntegerTypes.py* was removed):<br>
test/types/AbstractBase.py<br>
test/types/HideTestFailures.py<br>
test/types/TestFloatTypes.py<br>
test/types/TestFloatTypesExpr.py<br>
test/types/TestIntegerTypesExpr.py<br>
test/types/TestRecursiveTypes.py<br>
<br>
Tests were run in the lldb/test directory using the following commands:<br>
dotest:<br>
./dotest.py -v<br>
dosep:<br>
./dosep.py -s --options "-v"<br>
<br>
Comparing the test case totals, dotest correctly counts 46, but dosep counts only 16:<br>
dotest:<br>
Ran 46 tests in 75.934s<br>
dosep:<br>
Testing: 23 tests, 4 threads ## note: this number changes randonmly<br>
Ran 6 tests in 7.049s<br>
[PASSED TestFloatTypes.py] - 1 out of 23 test suites processed<br>
Ran 6 tests in 11.165s<br>
[PASSED TestFloatTypesExpr.py] - 2 out of 23 test suites processed<br>
Ran 30 tests in 54.581s ## FIXME: not counted?<br>
[PASSED TestIntegerTypesExpr.py] - 3 out of 23 test suites processed<br>
Ran 4 tests in 3.212s<br>
[PASSED TestRecursiveTypes.py] - 4 out of 23 test suites processed<br>
Ran 4 test suites (0 failed) (0.000000%)<br>
Ran 16 test cases (0 failed) (0.000000%)<br>
<br>
With test/types/TestIntegerTypesExpr.py* removed, both correctly count 16 test cases:<br>
dosep:<br>
Testing: 16 tests, 4 threads<br>
Ran 6 tests in 7.059s<br>
Ran 6 tests in 11.186s<br>
Ran 4 tests in 3.241s<br>
Ran 3 test suites (0 failed) (0.000000%)<br>
Ran 16 test cases (0 failed) (0.000000%)<br>
<br>
Note: I couldn't compare the test counts on all the tests because of the concern raised in <a href="http://reviews.llvm.org/rL237053" rel="noreferrer" target="_blank">http://reviews.llvm.org/rL237053</a>. That is, dotest can no longer complete the tests on OSX, as all test suites fail after test case 898: test_disassemble_invalid_vst_1_64_raw_data get ERRORs. I don't think that issue is related to problems in dosep.<br>
<br>
Thanks,<br>
-Dawn<br></div></div><span class="">
_______________________________________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a><br>
</span></blockquote></div></div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">-Todd</div></div>
</div></div>