[lldb-dev] Test suite rebuilding test executables many times

[Jim Ingham via lldb-dev]

It is fairly common practice (at least it is for me) when figuring out why a test failed, or adding to a test case, or when looking for a good example file to poke at, etc, to go to some relevant test directory, do a "make" then poke around a bunch.  I don't generally remember to clean when I'm done.  If the test suite didn't do make clean before running the tests then I'd get whatever state I left the binaries in after that investigation.  So I prefer doing make clean the first time you run a test in a given directory, but I have no objection to trying not to do the clean on subsequent tests in the same directory.  Also we do "dsym" and then "non-dsym" builds in the same directory on OS X, so we'd have to make sure that we clean when switching back & forth between the two kinds of tests, or we will leave a dSYM around at some point and stop testing .o file debugging.  Now that support is coming in for "dwo" debugging on Linux, we probably should also add the ability to test normal & dwo debugging there as well.  So this soon won't be just an OS X oddity...

Finally, there are some tests that rebuild the binaries on purpose - sadly I don't remember which ones.  If we're lucky they would fail if you switched the default and you could go fix them, but if you are unlucky they would succeed without actually testing what they were supposed to test.  So a little care would be needed to find these.

Jim

> On Aug 25, 2015, at 4:52 PM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> Another possibility is changing the arguments to buildDwarf and buildDsym.  Currently they take a clean argument with a default value of True.  Does this really need to be True?  If this were False by default it could drastically speed up the test suite.  And I can't think of a reason why make clean would need to run by default, because tear down is going to have to clean up the files manually anyway
> 
> On Tue, Aug 25, 2015 at 4:33 PM Zachary Turner <zturner at google.com> wrote:
> While looking into a Windows-specific issue involving TestTargetAPI.py, I noticed that we are building the exact same executable many times.  Every single test has a line such as self.buildDwarf() or self.buildDsym().  Those functions will first run make clean and then run make, essentially rebuilding the exact same program.
> 
> Is this necessary for some reason?  Each test suite already supports suite-specific setup and tear down by implementing a suite-specific setUp and tearDown function.  Any particular reason we can't build the executables a single time in setUp and clean them a single time in tearDown?
> 
> I don't think we need to retro-actively do this for every single test suite as it would be churn, but in a couple of places it would actually fix test failures on Windows, and improve performance of the test suite as a side benefit (as a result of reducing the number of compilations that need to happen)
> 
> Thoughts?
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_lldb-2Ddev&d=BQIGaQ&c=eEvniauFctOgLOKGJOplqw&r=aTCVT7yw0RLKhx7ZXY2faboS3m1dhXpYF-Av4XoSGMU&m=Eycd1fp0-akP8p_gZ00FUIzJIUFwuS78MvYOL_c9s5s&s=Xm3Muzt0_AVzEn-Y2pSy-gsnQsAbeNAmeo10Vr1VeT8&e=