<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Jul 22, 2014 at 5:01 AM, Matthew Gardiner <span dir="ltr"><<a href="mailto:mg11@csr.com" target="_blank">mg11@csr.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Folks,<br>
<br>
I've managed to tweak things (i.e. get llvm/lldb .sos and python site-packages under build/lib) in order that I can run dotest.py as follows:<br>
<br></blockquote><div><br></div><div>Great!</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
$ /mydir/llvm/tools/lldb/test<br>
<br>
LD_LIBRARY_PATH=/mydir/build/<u></u>lib/ PYTHONPATH=/mydir/build/lib/<u></u>python2.7/site-packages/ python dotest.py --executable=/mydir/build/bin/<u></u>lldb -v -l --compiler=gcc -q .<br>
<br></blockquote><div><br></div><div>Okay, glad you found a command line that worked. I'll get a FC VM up and work on a fix for that environment.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I note that the results say:<br>
<br>
Ran 1083 tests in 633.125s<br>
<br></blockquote><div><br></div><div>Reasonable - how many cores are you using? (This was a VM, right?)</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
FAILED (failures=1, skipped=619, expected failures=54, unexpected successes=15)<br>
<br>
Since I've only just managed to get the tests working, are the above results reasonable?<br>
<br></blockquote><div><br></div><div>Yes, that's reasonable for Linux. The skipped are generally Darwin/MacOSX tests --- there are nearly 2 tests total for every one that can run on Linux. The others are generally a variant of debuginfo packaging that is only available on MacOSX.</div>
<div><br></div><div>The expected failures represent the tests that we don't have working on Linux (often paired with FreeBSD) that are code and/or test bugs that need to be addressed. (If you're ever just feeling like doing some LLDB spelunking, these are great learning opportunities for one to pick up!)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
That is are expected failures=54, unexpected successes=15 ok?<br>
<br></blockquote><div><br></div><div>The unexpected successes represent one of two things:</div><div><br></div><div>1. tests marked XFAIL that are intermittent, and so sometimes pass, falling into this bucket. This is the best we can do with these for now until we get rid of the intermittent nature of the test. Note the multi-core test running that the build systems do stress the tests more heavily than when they are run individually.</div>
<div><br></div><div>2. tests marked XFAIL that always pass now, which should no longer be marked XFAIL. The majority do not fall into this category, but it does represent a state that can occur when we do fix the underlying race and/or timing issue that caused it to be intermittent in the first place.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The only actual failure I saw was:<br>
<br>
FAIL: test_stdcxx_disasm (TestStdCXXDisassembly.<u></u>StdCXXDisassembleTestCase)<br>
Do 'disassemble' on each and every 'Code' symbol entry from the std c++ lib.<br>
<br></blockquote><div><br></div><div>This is really the nugget of result your test run is showing. I'm not entirely sure why that one is failing. It could be a legitimate failure with changes in your code, or it could be something that surfaces in FC 20 that doesn't elsewhere. The test run should have made a directory called "lldb-test-traces". They go in different places depending on ninja vs. make builds. In ninja builds, it will be in your root build dir. In make builds it will be in the {my-build-dir}/tools/lldb/test dir. In that directory, you get a trace log file (*) for every test run that did not succeed - either because it was skipped, it failed (i.e. test assertion failed), it had an error (i.e. it failed but not because of an assert - something happened that was entirely unexpected like an i/o issue, seg fault, etc.), or it unexpectedly passed - marked xfail but succeeded. So - you should have a file called something like "Failed*test_stdcxx_disasm*.log" in that directory. You could look at the contents of that and see what failed.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I guess as I run the tests more often, I'll get more of a feel for it, but I just wondered if the above was a reasonable baseline.<br>
<br></blockquote><div><br></div><div>Generally the tests are in a state where if it fails, that represents an issue. I've spent quite a bit of time trying to get the test suite into that state, so that an issue represents a real problem. In your case, it could be a FC environment issue where it is really a test that - for that environment - is just not going to ever pass. In which case we need to either fix it or annotate it as a known issue and file a bug for it. For your particular case, the way to figure that out is to do a build and a test run against a clean-slate top of tree sync (essentially shelve any changes you have locally) and see what a clean-slate test run produces. If you always see that error, it's a tip-off that the test is broken in your environment.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
All tips/feedback welcome,<br>
Matt<br>
<br></blockquote><div><br></div><div>Happy testing!</div><div><br></div><div>-Todd</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Member of the CSR plc group of companies. CSR plc registered in England and Wales, registered number 4187346, registered office Churchill House, Cambridge Business Park, Cowley Road, Cambridge, CB4 0WZ, United Kingdom<br>
More information can be found at <a href="http://www.csr.com" target="_blank">www.csr.com</a>. Keep up to date with CSR on our technical blog, <a href="http://www.csr.com/blog" target="_blank">www.csr.com/blog</a>, CSR people blog, <a href="http://www.csr.com/people" target="_blank">www.csr.com/people</a>, YouTube, <a href="http://www.youtube.com/user/CSRplc" target="_blank">www.youtube.com/user/CSRplc</a>, Facebook, <a href="http://www.facebook.com/pages/CSR/191038434253534" target="_blank">www.facebook.com/pages/CSR/<u></u>191038434253534</a>, or follow us on Twitter at <a href="http://www.twitter.com/CSR_plc" target="_blank">www.twitter.com/CSR_plc</a>.<br>
New for 2014, you can now access the wide range of products powered by aptX at <a href="http://www.aptx.com" target="_blank">www.aptx.com</a>.<br>
______________________________<u></u>_________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@cs.uiuc.edu" target="_blank">lldb-dev@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev" target="_blank">http://lists.cs.uiuc.edu/<u></u>mailman/listinfo/lldb-dev</a><br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"><font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</div></div>