[lldb-dev] Test suite rebuilding test executables many times

[Zachary Turner via lldb-dev]

The first and second issues (cleaning once at startup, switching between
dsym and dwarf tests) can probably both be solved at the same time by
having the test runner sort the runs and do all dsym tests first, and then
all dwarf tests, and having TestBase do make clean once before each of
those steps.  What do you think?

I'm going to do some timings tomorrow to see how much faster the test suite
is when clean=False is the default.  I already confirmed that it fixes all
the failures I'm seeing though, so as long as it's agreeable I'd like to
make this change.

I'll wait and see if anyone can remember which tests rebuild binaries on
purpose.  Otherwise I will try to look through them and see if I can figure
it out.

On Tue, Aug 25, 2015 at 5:06 PM Jim Ingham <jingham at apple.com> wrote:

> It is fairly common practice (at least it is for me) when figuring out why
> a test failed, or adding to a test case, or when looking for a good example
> file to poke at, etc, to go to some relevant test directory, do a "make"
> then poke around a bunch.  I don't generally remember to clean when I'm
> done.  If the test suite didn't do make clean before running the tests then
> I'd get whatever state I left the binaries in after that investigation.  So
> I prefer doing make clean the first time you run a test in a given
> directory, but I have no objection to trying not to do the clean on
> subsequent tests in the same directory.  Also we do "dsym" and then
> "non-dsym" builds in the same directory on OS X, so we'd have to make sure
> that we clean when switching back & forth between the two kinds of tests,
> or we will leave a dSYM around at some point and stop testing .o file
> debugging.  Now that support is coming in for "dwo" debugging on Linux, we
> probably should also add the ability to test normal & dwo debugging there
> as well.  So this soon won't be just an OS X oddity...
>
> Finally, there are some tests that rebuild the binaries on purpose - sadly
> I don't remember which ones.  If we're lucky they would fail if you
> switched the default and you could go fix them, but if you are unlucky they
> would succeed without actually testing what they were supposed to test.  So
> a little care would be needed to find these.
>
> Jim
>
> > On Aug 25, 2015, at 4:52 PM, Zachary Turner via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
> >
> > Another possibility is changing the arguments to buildDwarf and
> buildDsym.  Currently they take a clean argument with a default value of
> True.  Does this really need to be True?  If this were False by default it
> could drastically speed up the test suite.  And I can't think of a reason
> why make clean would need to run by default, because tear down is going to
> have to clean up the files manually anyway
> >
> > On Tue, Aug 25, 2015 at 4:33 PM Zachary Turner <zturner at google.com>
> wrote:
> > While looking into a Windows-specific issue involving TestTargetAPI.py,
> I noticed that we are building the exact same executable many times.  Every
> single test has a line such as self.buildDwarf() or self.buildDsym().
> Those functions will first run make clean and then run make, essentially
> rebuilding the exact same program.
> >
> > Is this necessary for some reason?  Each test suite already supports
> suite-specific setup and tear down by implementing a suite-specific setUp
> and tearDown function.  Any particular reason we can't build the
> executables a single time in setUp and clean them a single time in tearDown?
> >
> > I don't think we need to retro-actively do this for every single test
> suite as it would be churn, but in a couple of places it would actually fix
> test failures on Windows, and improve performance of the test suite as a
> side benefit (as a result of reducing the number of compilations that need
> to happen)
> >
> > Thoughts?
> > _______________________________________________
> > lldb-dev mailing list
> > lldb-dev at lists.llvm.org
> >
> https://urldefense.proofpoint.com/v2/url?u=http-3A__lists.llvm.org_cgi-2Dbin_mailman_listinfo_lldb-2Ddev&d=BQIGaQ&c=eEvniauFctOgLOKGJOplqw&r=aTCVT7yw0RLKhx7ZXY2faboS3m1dhXpYF-Av4XoSGMU&m=Eycd1fp0-akP8p_gZ00FUIzJIUFwuS78MvYOL_c9s5s&s=Xm3Muzt0_AVzEn-Y2pSy-gsnQsAbeNAmeo10Vr1VeT8&e=
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150826/d29cb5e2/attachment.html>