<div dir="ltr">Awesome; thank you for the full summary, Todd! My take-homes from this are a) always run the full test suite on Linux, and b) don't worry about the test coverage & already broken tests, as it's being worked on.<div>
<br></div><div>I had thought about taking a detour to work on some of them, but it's not where I most want to be working in the code base, so I'm glad to have an excuse not to :-}.<br></div><div><div><br></div><div>
-- Randy</div><div><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 7, 2014 at 12:57 PM, Todd Fiala <span dir="ltr"><<a href="mailto:tfiala@google.com" target="_blank">tfiala@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And now to answer both of your actual questions :-P<div class=""><div><br></div><div>> <span style="font-family:arial,sans-serif;font-size:13px">* What is the expectation as to which tests will be run before submission of patches?</span></div>
<div><br></div></div><div>My take is you should run tests for your platform at a bare minimum. So, if you're developing on Linux, then run 'ninja check-lldb' or the equivalent, on your platform, and there should be no new tests showing up in the list of failed tests. (In an ideal world, which we'll get too soon-ish, there should be no tests failing here, period, but as it stands per earlier in the email thread, there are some failing right now).</div>
<div><br></div><div>Right now this will just run local tests on your platform. I'm okay with having the build bots handle the remote test runs.</div><div><br></div><div>If tests break on other platforms after submitting a test, the expectation I have is that the author work to get those resolved quickly or we revert the change until we can work out the right fix/change for the broken platforms (which might include reworking the change that triggered the break).</div>
<div class="">
<div><br></div><div>> <span style="font-family:arial,sans-serif;font-size:13px">* What is the current status for the Linux tests? Is anyone already looking at bringing the test coverage to parity with Darwin?</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div></div><div><span style="font-family:arial,sans-serif;font-size:13px">That's the bit I was answering with my first two posts.</span></div>
</div><div class="HOEnZb"><div class="h5">
<div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 7, 2014 at 9:20 AM, Todd Fiala <span dir="ltr"><<a href="mailto:tfiala@google.com" target="_blank">tfiala@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">I should also add that as of this moment, we have a several tests (5 or 6 on Linux) that fail intermittently on Linux. Some of them are due to the multithreaded test running and attempts by multiple tests to grab resources (like network ports) without any type of mediator. We'll be addressing these in the near future. I haven't flipped them to skipped since they tend to run fine in solo mode (running a test at a time), but realistically we need to either fix them for multithreaded or skip them if they're being intermittent under load/resource-contention issues.<div>
<br></div><div>MacOSX has more tests failing right now (last I checked more like 10+). Also, MacOSX tests are not running parallel for some reason we haven't yet fully diagnosed (but there is a bugzilla bug on it). I went so far as to see the thread queues for running the tests were all waiting on the python subprocess calls appropriately, but only one at a time was actually getting through. We'll want to get those tests running in parallel so that they're more agreeable to run - on Linux I can run the test suite in well under 2 minutes, whereas on MacOSX the combination of fewer cores and no multithreading means I take 20 - 45 minutes to run the tests. I'm sure that's a significant disincentive for running the tests there.</div>
<div><br></div><div><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jul 7, 2014 at 8:47 AM, Todd Fiala <span dir="ltr"><<a href="mailto:tfiala@google.com" target="_blank">tfiala@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Randy,<div><br></div><div>Great questions. See a few threads we had on this a while back when I started looking into this:</div>
<div><br></div><div><a href="http://lists.cs.uiuc.edu/pipermail/lldb-dev/2014-March/003470.html" target="_blank">http://lists.cs.uiuc.edu/pipermail/lldb-dev/2014-March/003470.html</a><br>
</div><div><a href="http://lists.cs.uiuc.edu/pipermail/lldb-dev/2014-March/003474.html" target="_blank">http://lists.cs.uiuc.edu/pipermail/lldb-dev/2014-March/003474.html</a><br></div><div><div><br></div><div>> <span style="font-family:arial,sans-serif;font-size:12.727272033691406px">I'm seeing a lot of "UNSUPPORTED" tests when I run on Linux and that makes me wonder how much test coverage I'm actually getting if I run those tests before submitting patches</span><span style="font-family:arial,sans-serif;font-size:12.727272033691406px"> </span></div>
<div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px"><br></span></div></div><div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">Roughly half the tests won't run under Linux on the local test suite. MacOSX has two different mechanisms for handling debug info packaging, so they test a large swath of lldb two ways, one of which isn't appropriate for Linux. Those will all show as skipped.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:12.727272033691406px">Back in March I did a few things on Linux:</span></div>
<div><ul><li><font face="arial, sans-serif">Went through all the tests that were entirely disabled. I turned them on, and if they made sense to run at all on Linux, I saw if they either always failed or always succeeded. If they did, I turned them on in the normal state (where they are expected to pass). Those that always failed I marked as XFAIL (expected failure). Those that passed intermittently for me I marked as skipped to avoid noise and ensured there was at least one bug tracking them in bugzilla.</font></li>
<li><font face="arial, sans-serif">Enabled code coverage for the local Linux test run. As expected, much of the remote functionality wasn't covered, but a non-trivial amount of the code was hit by the tests running on Linux. When I eventually get a build bot up, I'm hoping to include a code coverage run for local tests, so we can get an idea of which code is tested by the local test suite. (Soon I will have lldb-platform running for Linux x86-based, in which case we can run the remote suite as well).</font></li>
</ul><div>My goal at the time was to find out just how bad the state of the Linux code was w/r/t the MacOSX side. It ended up being not quite as bad as I expected due to some of that info I published.</div></div><div><br>
</div><div>I have several people on my team now that are working their way through the bug database and will be making a point of correcting the more egregious issues on Linux. We're working our way towards getting Android working via the lldb-gdbserver (llgs) remote support.</div>
<div><br></div><div>Hope that helps!</div><div><br></div><div>Sincerely,</div><div>Todd Fiala</div></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Sun, Jul 6, 2014 at 6:46 PM, Randy Smith <span dir="ltr"><<a href="mailto:rdsmith@chromium.org" target="_blank">rdsmith@chromium.org</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><br><div>Two questions:</div><div>* What is the expectation as to which tests will be run before submission of patches?</div>
<div>* What is the current status for the Linux tests? Is anyone already looking at bringing the test coverage to parity with Darwin?</div>
<div><br></div><div>I'm seeing a lot of "UNSUPPORTED" tests when I run on Linux and that makes me wonder how much test coverage I'm actually getting if I run those tests before submitting patches (though of the 616 UNSUPPORTED tests on Linux, all but 186 seem to involve dsym, which is probably ok to be unsupported on Linux). So I'm wondering about what the expectations are for test runs before patch submission, and how easily I can meet those expectations if my primary work environment is Linux.</div>
<div><br></div><div>The context is that I'm trying to learn enough about lldb to make real contributions, and I'm doing that by first trying to use it and make it work well as my primary debugger in my usual work environment (Linux targeting chromium). </div>
<div><br></div><div>Thanks in advance for any enlightenment ...</div><span><font color="#888888"><div><br></div><div>-- Randy Smith</div><div><br></div><div><br></div></font></span></div>
<br></div></div>_______________________________________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@cs.uiuc.edu" target="_blank">lldb-dev@cs.uiuc.edu</a><br>
<a href="http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev" target="_blank">http://lists.cs.uiuc.edu/mailman/listinfo/lldb-dev</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'">
<tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"><font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"><font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><table cellspacing="0" cellpadding="0" style="color:rgb(136,136,136);font-family:'Times New Roman'"><tbody><tr style="color:rgb(85,85,85);font-family:sans-serif;font-size:small">
<td nowrap style="border-top-style:solid;border-top-color:rgb(213,15,37);border-top-width:2px">Todd Fiala |</td><td nowrap style="border-top-style:solid;border-top-color:rgb(51,105,232);border-top-width:2px"> Software Engineer |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(0,153,57);border-top-width:2px"> <a href="mailto:tfiala@google.com" style="color:rgb(17,85,204)" target="_blank"><span style="background-color:rgb(255,255,204);color:rgb(34,34,34);background-repeat:initial initial">tfiala@google.com</span></a> |</td>
<td nowrap style="border-top-style:solid;border-top-color:rgb(238,178,17);border-top-width:2px"><font color="#1155cc"> <a>650-943-3180</a></font></td></tr></tbody></table><br></div>
</div>
</div></div></blockquote></div><br></div>