<div dir="ltr">Hmm, yeah it looks like it did the rerun and then after finishing the rerun, it's just hanging.<div><br></div><div>Let's have a look right after r255676 goes through this builder. I hit a hang in the curses output display due to the recursive taking of a lock on a lock that was not recursive-enabled. While I would have expected to see that with the basic results output that this builder here is using when I was testing earlier, it's possible somehow that we're hitting a path here that is attempting to recursively take a lock.</div><div><br></div><div>Do you know if it is happening every single time a rerun occurs? (Hopefully yes?)</div><div><br></div><div>-Todd</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 15, 2015 at 1:38 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Yep, I'll have a look!</div><div class="gmail_extra"><div><div class="h5"><br><div class="gmail_quote">On Tue, Dec 15, 2015 at 12:43 PM, Ying Chen <span dir="ltr"><<a href="mailto:chying@google.com" target="_blank">chying@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Todd,<div><br></div><div>It is noticed on lldb android builders that the test_runner didn't exit after rerun, which caused buildbot timeout since the process was hanging for over 20 minutes.</div><div>Could you please take a look if that's related to your change?</div><div><br></div><div>Please see the following builds. </div><div><a href="http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio" target="_blank">http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test3/logs/stdio</a><br></div><div><a href="http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio" target="_blank">http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-android/builds/4305/steps/test7/logs/stdio</a><br></div><div><br></div><div>Thanks,</div><div>Ying</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 4:52 PM, Todd Fiala via lldb-dev <span dir="ltr"><<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And, btw, this shows the rerun logic working (via the --rerun-all-issues flag):<div><br></div><div><div>time test/dotest.py --executable `pwd`/build/Debug/lldb --threads 24 --rerun-all-issues</div><div>Testing: 416 test suites, 24 threads</div><div>377 out of 416 test suites processed - TestSBTypeTypeClass.py</div><div><br></div><div>Session logs for test failures/errors/unexpected successes will go into directory '2015-12-14-16_44_28'</div><div>Command invoked: test/dotest.py --executable /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior -p TestMultithreaded.py /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test --event-add-entries worker_index=3:int</div><div><br></div><div>Configuration: arch=x86_64 compiler=clang</div><div>----------------------------------------------------------------------</div><div>Collected 8 tests</div><div><br></div><div>lldb_codesign: no identity found</div><div>lldb_codesign: no identity found</div><div>lldb_codesign: no identity found</div><div>lldb_codesign: no identity found</div><div>lldb_codesign: no identity found</div><div>lldb_codesign: no identity found</div><div>lldb_codesign: no identity found</div><div><br></div><div>[TestMultithreaded.py FAILED]</div><div>Command invoked: /usr/bin/python test/dotest.py --executable /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior -p TestMultithreaded.py /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test --event-add-entries worker_index=3:int</div><div>396 out of 416 test suites processed - TestMiBreak.py</div><div><br></div><div>Session logs for test failures/errors/unexpected successes will go into directory '2015-12-14-16_44_28'</div><div>Command invoked: test/dotest.py --executable /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior -p TestDataFormatterObjC.py /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test --event-add-entries worker_index=12:int</div><div><br></div><div>Configuration: arch=x86_64 compiler=clang</div><div>----------------------------------------------------------------------</div><div>Collected 26 tests</div><div><br></div><div><br></div><div>[TestDataFormatterObjC.py FAILED]</div><div>Command invoked: /usr/bin/python test/dotest.py --executable /Users/tfiala/src/lldb-tot/lldb/build/Debug/lldb --threads 24 --rerun-all-issues -s 2015-12-14-16_44_28 --results-port 62322 --inferior -p TestDataFormatterObjC.py /Users/tfiala/src/lldb-tot/lldb/packages/Python/lldbsuite/test --event-add-entries worker_index=12:int</div><div>416 out of 416 test suites processed - TestLldbGdbServer.py</div><div>2 test files marked for rerun</div><div><br></div><div><br></div><div>Rerunning the following files:</div><div> functionalities/data-formatter/data-formatter-objc/TestDataFormatterObjC.py</div><div> api/multithreaded/TestMultithreaded.py</div><div>Testing: 2 test suites, 1 thread</div><div>2 out of 2 test suites processed - TestMultithreaded.py</div><div>Test rerun complete</div><div><br></div><div><br></div><div>=============</div><div>Issue Details</div><div>=============</div><div>UNEXPECTED SUCCESS: test_symbol_name_dsym (functionalities/completion/TestCompletion.py)</div><div>UNEXPECTED SUCCESS: test_symbol_name_dwarf (functionalities/completion/TestCompletion.py)</div><div><br></div><div>===================</div><div>Test Result Summary</div><div>===================</div><div>Test Methods: 1695</div><div>Reruns: 30</div><div>Success: 1367</div><div>Expected Failure: 90</div><div>Failure: 0</div><div>Error: 0</div><div>Exceptional Exit: 0</div><div>Unexpected Success: 2</div><div>Skip: 236</div><div>Timeout: 0</div><div>Expected Timeout: 0</div></div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 4:51 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And that fixed the rest as well. Thanks, Siva!<div><br></div><div>-Todd</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 4:44 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Heh you were skinning the same cat :-)<div><br></div><div>That fixed the one I was just looking at, running the others now.</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 4:42 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Yep, will try now... (I was just looking at the condition testing logic since it looks like something isn't quite right there).</div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 4:39 PM, Siva Chandra <span dir="ltr"><<a href="mailto:sivachandra@google.com" target="_blank">sivachandra@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can you try again after taking my change at r255584?<br>
<div><div><br>
On Mon, Dec 14, 2015 at 4:31 PM, Todd Fiala via lldb-dev<br>
<<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>> wrote:<br>
> I'm having some of these blow up.<br>
><br>
> In the case of test/lang/c/typedef/Testtypedef.py, it looks like some of the<br>
> @expected decorators were changed a bit, and perhaps they are not pound for<br>
> pound the same. For example, this test used to really be marked XFAIL (via<br>
> an expectedFailureClang directive), but it looks like the current marking of<br>
> compiler="clang" is either not right or not working, since the test is run<br>
> on OS X and is treated like it is expected to pass.<br>
><br>
> I'm drilling into that a bit more, that's just the first of several that<br>
> fail with these changes on OS X.<br>
><br>
> On Mon, Dec 14, 2015 at 3:03 PM, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> wrote:<br>
>><br>
>> I've checked in r255567 which fixes a problem pointed out by Siva. It<br>
>> doesn't sound related to in 255542, but looking at those logs I can't really<br>
>> tell how my CL would be related. If r255567 doesn't fix the bots, would<br>
>> someone mind helping me briefly? r255542 seems pretty straightforward, so I<br>
>> don't see why it would have an effect here.<br>
>><br>
>> On Mon, Dec 14, 2015 at 2:35 PM Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>> wrote:<br>
>>><br>
>>> Ah yes I see. Thanks, Ying (and Siva! Saw your comments too).<br>
>>><br>
>>> On Mon, Dec 14, 2015 at 2:34 PM, Ying Chen <<a href="mailto:chying@google.com" target="_blank">chying@google.com</a>> wrote:<br>
>>>><br>
>>>> Seems this is the first build that fails, and it only has one CL 255542.<br>
>>>><br>
>>>> <a href="http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446" rel="noreferrer" target="_blank">http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9446</a><br>
>>>> I believe Zachary is looking at that problem.<br>
>>>><br>
>>>> On Mon, Dec 14, 2015 at 2:18 PM, Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>><br>
>>>> wrote:<br>
>>>>><br>
>>>>> I am seeing several failures on the Ubuntu 14.04 testbot, but<br>
>>>>> unfortunately there are a number of changes that went in at the same time on<br>
>>>>> that build. The failures I'm seeing are not appearing at all related to the<br>
>>>>> test running infrastructure.<br>
>>>>><br>
>>>>> Anybody with a fast Linux system able to take a look to see what<br>
>>>>> exactly is failing there?<br>
>>>>><br>
>>>>> -Todd<br>
>>>>><br>
>>>>> On Mon, Dec 14, 2015 at 1:39 PM, Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>><br>
>>>>> wrote:<br>
>>>>>><br>
>>>>>> Hi all,<br>
>>>>>><br>
>>>>>> I just put in the single-worker, low-load, follow-up test run pass in<br>
>>>>>> r255543. Most of the work for it went in late last week, this just mostly<br>
>>>>>> flips it on.<br>
>>>>>><br>
>>>>>> The feature works like this:<br>
>>>>>><br>
>>>>>> * First test phase works as before: run all tests using whatever level<br>
>>>>>> of concurrency is normally used. (e.g. 8 works on an 8-logical-core box).<br>
>>>>>><br>
>>>>>> * Any timeouts, failures, errors, or anything else that would have<br>
>>>>>> caused a test failure is eligible for rerun if either (1) it was marked as a<br>
>>>>>> flakey test via the flakey decorator, or (2) if the --rerun-all-issues<br>
>>>>>> command line flag is provided.<br>
>>>>>><br>
>>>>>> * After the first test phase, if there are any tests that met rerun<br>
>>>>>> eligibility that would have caused a test failure, those get run using a<br>
>>>>>> serial test phase. Their results will overwrite (i.e. replace) the previous<br>
>>>>>> result for the given test method.<br>
>>>>>><br>
>>>>>> The net result should be that tests that were load sensitive and<br>
>>>>>> intermittently fail during the first higher-concurrency test phase should<br>
>>>>>> (in theory) pass in the second, single worker test phase when the test suite<br>
>>>>>> is only using a single worker. This should make the test suite generate<br>
>>>>>> fewer false positives on test failure notification, which should make<br>
>>>>>> continuous integration servers (testbots) much more useful in terms of<br>
>>>>>> generating actionable signals caused by version control changes to the lldb<br>
>>>>>> or related sources.<br>
>>>>>><br>
>>>>>> Please let me know if you see any issues with this when running the<br>
>>>>>> test suite using the default output. I'd like to fix this up ASAP. And for<br>
>>>>>> those interested in the implementation, I'm happy to do post-commit<br>
>>>>>> review/changes as needed to get it in good shape.<br>
>>>>>><br>
>>>>>> I'll be watching the builders now and will address any issues as I<br>
>>>>>> see them.<br>
>>>>>><br>
>>>>>> Thanks!<br>
>>>>>> --<br>
>>>>>> -Todd<br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>><br>
>>>>> --<br>
>>>>> -Todd<br>
>>>><br>
>>>><br>
>>><br>
>>><br>
>>><br>
>>> --<br>
>>> -Todd<br>
><br>
><br>
><br>
><br>
> --<br>
> -Todd<br>
><br>
</div></div>> _______________________________________________<br>
> lldb-dev mailing list<br>
> <a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a><br>
><br>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
<br>_______________________________________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">-Todd</div></div>
</div>