[lldb-dev] Test suite rebuilding test executables many times

[Zachary Turner via lldb-dev]

I thought of this too and I started prototyping it.

The issue that I ran into is that dsym and dwarf tests can all be xfailed,
skipped, etc for different reasons, so if there is one method body, you
need a way to still define the set of conditions under which dsym and dwarf
tests should run, skip, xfail, timeout, etc.

Do you want to start writing @skipIfDwarfAndOsIsLinuxButCompilerIsNotClang?
 Because I know I don't want to deal with the combinatorial explosion of
decorators that would result :)

I have some ideas here as well, for example I think we only actually need 1
decorator that we can configure via keyword arguments that can handle
arbitrarily complex scenarios of xfailing, debug infos, etc.  e.g.
@lldb_test(debug_types="dwo,dwarf", xfail = {...}, skip = {...}).  But it's
all unrelated to the original problem I'm trying to solve.  So I think it
would be good to design a solution for that, but to do it separately.

The nice thing about just changing the default from clean to not clean is
that it's about a 3 line change, has potentially large speed improvements
across the board, and also fixes bugs.



On Wed, Aug 26, 2015 at 2:01 AM Pavel Labath <labath at google.com> wrote:

> On 26 August 2015 at 06:14, Zachary Turner via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
> >
> > I'll wait and see if anyone can remember which tests rebuild binaries on
> > purpose.  Otherwise I will try to look through them and see if I can
> figure
> > it out.
>
> TestInferiorChanged is one that I remember.
>
> I think this is a good thing to do, but it will need to be done with a
> steady hand.
>
>
>
> I was also thinking about the dsym/dwo tests.. Instead of basically
> having a copy of each test for dwarf and dsym (and soon also dwo), how
> about having just one test, and have some higher level logic (the test
> runner) know that it needs to execute each test multiple times. The
> tests would then just do a buildDefault() (or something) and on the
> first run it would build normal dwarf, on the second one dsym, etc. If
> we need to run a test only for some combination of debug infos, we
> could have @skipIfDsym annotations, like we do for the rest of stuff.
> I think this will make what Zachary is proposing easier to do, and it
> will make the test writing less awkard.
>
>
> What do you think? I'm ready to chip in on this if we agree to go down
> this way...
>
> pl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150826/46f3a69d/attachment.html>