<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 15, 2015 at 8:50 AM, Adrian McCarthy via lldb-dev <span dir="ltr"><<a href="mailto:lldb-dev@lists.llvm.org" target="_blank">lldb-dev@lists.llvm.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I've tracked down a source of flakiness in tests on Windows to Python object lifetimes and the SB interface, and I'm wondering how best to handle it.</div><div><br></div><div>Consider this portion of a test from TestTargetAPI:</div><div><br></div><div><font face="monospace, monospace" size="1"> def find_functions(self, exe_name):</font></div><div><font face="monospace, monospace" size="1"> """Exercise SBTaget.FindFunctions() API."""</font></div><div><font face="monospace, monospace" size="1"> exe = os.path.join(os.getcwd(), exe_name)</font></div><div><font face="monospace, monospace" size="1"><br></font></div><div><font face="monospace, monospace" size="1"> # Create a target by the debugger.</font></div><div><font face="monospace, monospace" size="1"> target = self.dbg.CreateTarget(exe)</font></div><div><font face="monospace, monospace" size="1"> self.assertTrue(target, VALID_TARGET)</font></div><div><font face="monospace, monospace" size="1"> list = target.FindFunctions('c', lldb.eFunctionNameTypeAuto)</font></div><div><font face="monospace, monospace" size="1"> self.assertTrue(list.GetSize() == 1)</font></div><div><font face="monospace, monospace" size="1"><br></font></div><div><font face="monospace, monospace" size="1"> for sc in list:</font></div><div><font face="monospace, monospace" size="1"> self.assertTrue(sc.GetModule().GetFileSpec().GetFilename() == exe_name)</font></div><div><font face="monospace, monospace" size="1"> self.assertTrue(sc.GetSymbol().GetName() == 'c')</font></div><div><br></div><div>The local variables go out of scope when the function exits, but the SB (C++) objects they represent aren't (always) immediately destroyed. At least some of these objects keep references to the executable module in the shared module list, so when the test framework cleans up and calls `SBDebugger::DeleteTarget`, the module isn't orphaned, so LLDB maintains an open handle to the executable.</div><div><br></div><div>The result of the lingering handle is that, when the next test case in the test suite tries to re-build the executable, it fails because the file is not writable. (This is problematic on Windows because the file system works differently in this regard than Unix derivatives.) Every subsequent case in the test suite fails.</div><div><br></div><div>I managed to make the test work reliably by rewriting it like this:</div><div><br></div><div><font size="1" face="monospace, monospace"> def find_functions(self, exe_name):</font></div><div><font size="1" face="monospace, monospace"> """Exercise SBTaget.FindFunctions() API."""</font></div><div><font size="1" face="monospace, monospace"> exe = os.path.join(os.getcwd(), exe_name)</font></div><div><font size="1" face="monospace, monospace"><br></font></div><div><font size="1" face="monospace, monospace"> # Create a target by the debugger.</font></div><div><font size="1" face="monospace, monospace"> target = self.dbg.CreateTarget(exe)</font></div><div><font size="1" face="monospace, monospace"> self.assertTrue(target, VALID_TARGET)</font></div><div><font size="1" face="monospace, monospace"><br></font></div><div><font size="1" face="monospace, monospace"> try:</font></div><div><font size="1" face="monospace, monospace"> list = target.FindFunctions('c', lldb.eFunctionNameTypeAuto)</font></div><div><font size="1" face="monospace, monospace"> self.assertTrue(list.GetSize() == 1)</font></div><div><font size="1" face="monospace, monospace"><br></font></div><div><font size="1" face="monospace, monospace"> for sc in list:</font></div><div><font size="1" face="monospace, monospace"> try:</font></div><div><font size="1" face="monospace, monospace"> self.assertTrue(sc.GetModule().GetFileSpec().GetFilename() == exe_name)</font></div><div><font size="1" face="monospace, monospace"> self.assertTrue(sc.GetSymbol().GetName() == 'c')</font></div><div><font size="1" face="monospace, monospace"> finally:</font></div><div><font size="1" face="monospace, monospace"> del sc</font></div><div><font size="1" face="monospace, monospace"><br></font></div><div><font size="1" face="monospace, monospace"> finally:</font></div><div><font size="1" face="monospace, monospace"> del list</font></div><div><br></div><div>The finally blocks ensure that the corresponding C++ objects are destroyed, even if the function exits as a result of a Python exception (e.g., if one of the assertion expressions is false and the code throws an exception). Since the objects are destroyed, the reference counts are back to where they should be, and the orphaned module is closed when the target is deleted.</div><div><br></div><div>But this is ugly and maintaining it would be error prone. Is there a better way to address this?</div><div><br></div></div></blockquote><div><br></div><div>We should drill into why they're not cleaning up right away. If it's out of our control (i.e. some kind of lazy clean up), then maybe we can put in some meta-magic that generates the deletes like you added.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>In general, it seems bad that our tests aren't well-isolated. </div></div></blockquote><div><br></div><div>Each .py file is getting run as a separate process unless the tests are run in a way that disables the concurrent test runner. Methods within a given .py file are getting run in the same process. So we're really talking about test methods within a file, not test methods across files.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div> I sympathize with the concern that re-starting LLDB for each test case would slow down testing, but I'm also concerned that the state of LLDB for any given test case can depend on what happened in the earlier cases.</div><span class="HOEnZb"><font color="#888888"><div><br></div></font></span></div></blockquote><div><br></div><div>One thing I plan to do mid-term is:</div><div>* announce all the tests that can run in a python file</div><div>* ensure we report on tests *not* run because of a crash (i.e. get counts for true unknowns, not just info on tests that pass/fail/error).</div><div>* having an "announce-only" pass so we can dynamically collect up all he tests without necessarily running them. (for things like test runners that can list all the tests available and want to show progress).</div><div><br></div><div>This would conceivably support the underpinnings to say:</div><div>* collect all the tests (announce-only)</div><div>* add a way to fire up a single test method (we sortta have that already), and only do one test method per process.</div><div><br></div><div>I think there is additional coordination that would have to happen to handle inferior binaries build for a test and make sure they exist where (and when) they need to, with the right arch settings and whatnot.</div><div><br></div><div>But, that would definitely get full isolation per test method.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class="HOEnZb"><font color="#888888"><div></div><div>Adrian.</div></font></span></div>
<br>_______________________________________________<br>
lldb-dev mailing list<br>
<a href="mailto:lldb-dev@lists.llvm.org">lldb-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr">-Todd</div></div>
</div></div>