<div dir="ltr">I wonder if you need a flush somewhere before you invoke the cleanup func? Would that do it? It looks like the sending side of the connection is closing before the receiving side has received all its data.</div><br><div class="gmail_quote"><div dir="ltr">On Tue, Dec 15, 2015 at 3:49 PM Adrian McCarthy <<a href="mailto:amccarth@google.com">amccarth@google.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>With Todd's change, I was getting a Ninja crash. Zach and I replaced the returns Todd added with raises, in order to propagate the exception up the stack, and that avoids the Ninja crash, so I'll check that in in a moment.</div><div><br></div><div>In the mean time, here's the error message we got out of it.</div><div><br></div><div>155 out of 416 test suites processed - TestBacktraceAll.py</div><div>INFO: received socket error when reading data from test inferior:</div></div><div dir="ltr"><div>[Errno 10054] An existing connection was forcibly closed by the remote host</div></div><div dir="ltr"><div>error: uncaptured python exception, closing channel <lldbsuite.test.dotest_channels.UnpicklingForwardingReaderChannel connected 127.</div></div><div dir="ltr"><div>0.0.1:58961 at 0x2bb8878> (<class 'socket.error'>:[Errno 10054] An existing connection was forcibly closed by the remote host [D:\Python_for_lldb\x86\lib\asyncore.py|read|83] [D:\Python_for_lldb\x86\lib\asyncore.py|handle_read_event|449] [D:\src\llvm\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|137] [D:\Python_for_lldb\x86\lib\asyncore.py|recv|387])</div><div>175 out of 416 test suites processed - TestNoSuchArch.py</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 3:58 PM, Todd Fiala via lldb-dev <span dir="ltr"><<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Zachary,<div><br></div><div>I just put in:</div><div>r255581<br></div><div><br></div><div>which should hopefully:</div><div>(1) catch the exception you see there,</div><div>(2) handle it gracefully in the common and to-be-expected case of the test inferior going down hard, and</div><div>(3) print out an error if anything else unexpected is happening here.</div><div><br></div><div>Let me know if you get any more info with it. Thanks!</div><div><br></div><div>-Todd</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 2:16 PM, Todd Fiala <span dir="ltr"><<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Yeah that's a messed up exception scenario that is hard to read. I'll figure something out when I repro it. One side is closing before the other is expecting it, but likely in a way we need to expect.<div><br></div><div>I think it is ugly-ified because it is coming from some kind of worker thread within async-core.</div><div><br></div><div>I will get something in to help it today. The first bit may be just catching the exception as you mentioned.</div></div><div class="gmail_extra"><div><div><br><div class="gmail_quote">On Mon, Dec 14, 2015 at 2:05 PM, Zachary Turner <span dir="ltr"><<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">If nothing else, maybe we can print out a more useful exception backtrace. What kind of exception, what line, and what was the message? That might help give us a better idea of what's causing it.</div><div><div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 14, 2015 at 2:03 PM Todd Fiala <<a href="mailto:todd.fiala@gmail.com" target="_blank">todd.fiala@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Zachary!<div><br></div><div><br></div><div><br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote"></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Dec 14, 2015 at 1:28 PM, Zachary Turner via lldb-dev <span dir="ltr"><<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Todd, lately I've been seeing this sporadically when running the test suite.<div><br></div><div><div>[TestNamespaceLookup.py FAILED]</div><div>Command invoked: C:\Python27_LLDB\x86\python_d.exe D:\src\llvm\tools\lldb\test\dotest.pyc -q --arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe --results-port 55886 --inferior -p TestNamespaceLookup.py D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --event-add-entries worker_index=10:int</div><div>416 out of 416 test suites processed - TestAddDsymCommand.py error: uncaptured python exception, closing channel <lldbsuite.test.dotest_channels.UnpicklingForwardingReaderChannel connected <a href="http://127.0.0.1:56008" target="_blank">127.0.0.1:56008</a> at 0x2bdd578> (<class 'socket.error'>:[Errno 10054] An existing connection was forcibly closed by the remote host [C:\Python27_LLDB\x86\lib\asyncore.py|read|83] [C:\Python27_LLDB\x86\lib\asyncore.py|handle_read_event|449] [D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest_channels.py|handle_read|133] [C:\Python27_LLDB\x86\lib\asyncore.py|recv|387])</div></div><div><br></div><div>It seems to happen randomly and not always on the same test. Sometimes it doesn't happen at all. I wonder if this could be related to some of the work that's been going on recently. Are you seeing this? Any idea how to diagnose?</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Eww.</div><div><br></div><div>That *looks* like one side of the connection between the inferior and the test runner process choked on reading content from the test event socket when the other end went down. Reading it a bit more carefully, it looks like it is the event collector (which would be the parallel test runner side) that was receiving when the socket went down.</div><div><br></div><div>I think this means I just need to put a try block around the receiver and just bail out gracefully (possibly with a message) when that happens at an unexpected time. Since test inferiors can die at any time, possibly due to a timeout where they are forcibly killed, we do need to handle that gracefully.'</div><div><br></div><div>I'll see if I can force it, replicate it, and fix it. I'll look at that now (pending watching the buildbots for the other change I just put in).</div><div><br></div><div>And yes, this would be a code path that we use heavily with the xUnit reporter, but only started getting used by you more recently when I turned on the newer summary results by default. (The newer summary results use the test event system, which means test inferiors are now going to be using the sockets to pass back test events, where you didn't have that happening before unless you used the curses or xUnit results formatter).</div><div><br></div><div>I hope to have it reproduced and fixed up here quickly. I suspect you may have an environment that just might make it more prevalent, but it needs to be fixed.</div><div><br></div><div>Hopefully back in a bit with a fix!</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>_______________________________________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a><br>
<br></blockquote></div></div></div><div dir="ltr"><div class="gmail_extra"><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr">-Todd</div></div>
</div></div></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div><div dir="ltr">-Todd</div></div>
</font></span></div>
<br>_______________________________________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a><br>
<br></blockquote></div><br></div>
</blockquote></div>