[lldb-dev] Test suite rebuilding test executables many times

[Jim Ingham via lldb-dev]

That seems fair.  It would be great to have some higher level mechanism to generate runs of the tests that only differ by how the target programs are built.  For instance, if some day we ever get serious about optimized code debugging (suppresses giggle) it would be good to run the test suite on optimized binaries as well...  So if you are inclined to work on that, that would be excellent, and then we can revisit how to do clean etc. at that time.

Jim

> On Aug 26, 2015, at 11:10 AM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> I think I'm goign to abandon this idea for now.  Mostly because I've found a workaround which is a) one line of code and b) only affects windows.  So the impact of this workaround is narrower, no risk of messing up any tests which depend on cleaning, and gives more time to come up with a more comprehensive solution, such as something like I and Pavel proposed earlier.
> 
> On Wed, Aug 26, 2015 at 10:24 AM Todd Fiala <todd.fiala at gmail.com> wrote:
> Not sure if this is relevant, but I seem to recall the remote test execution would spin off each test method run (test case level, not test suite level) into a new directory.  I don't think that would be inherently broken by a no-clean scenario but we'd want to make sure it doesn't break.
> 
> -Todd
> 
> On Wed, Aug 26, 2015 at 7:52 AM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> I thought of this too and I started prototyping it.
> 
> The issue that I ran into is that dsym and dwarf tests can all be xfailed, skipped, etc for different reasons, so if there is one method body, you need a way to still define the set of conditions under which dsym and dwarf tests should run, skip, xfail, timeout, etc.  
> 
> Do you want to start writing @skipIfDwarfAndOsIsLinuxButCompilerIsNotClang?  Because I know I don't want to deal with the combinatorial explosion of decorators that would result :)
> 
> I have some ideas here as well, for example I think we only actually need 1 decorator that we can configure via keyword arguments that can handle arbitrarily complex scenarios of xfailing, debug infos, etc.  e.g. @lldb_test(debug_types="dwo,dwarf", xfail = {...}, skip = {...}).  But it's all unrelated to the original problem I'm trying to solve.  So I think it would be good to design a solution for that, but to do it separately.
> 
> The nice thing about just changing the default from clean to not clean is that it's about a 3 line change, has potentially large speed improvements across the board, and also fixes bugs.
> 
> 
> 
> On Wed, Aug 26, 2015 at 2:01 AM Pavel Labath <labath at google.com> wrote:
> On 26 August 2015 at 06:14, Zachary Turner via lldb-dev
> <lldb-dev at lists.llvm.org> wrote:
> >
> > I'll wait and see if anyone can remember which tests rebuild binaries on
> > purpose.  Otherwise I will try to look through them and see if I can figure
> > it out.
> 
> TestInferiorChanged is one that I remember.
> 
> I think this is a good thing to do, but it will need to be done with a
> steady hand.
> 
> 
> 
> I was also thinking about the dsym/dwo tests.. Instead of basically
> having a copy of each test for dwarf and dsym (and soon also dwo), how
> about having just one test, and have some higher level logic (the test
> runner) know that it needs to execute each test multiple times. The
> tests would then just do a buildDefault() (or something) and on the
> first run it would build normal dwarf, on the second one dsym, etc. If
> we need to run a test only for some combination of debug infos, we
> could have @skipIfDsym annotations, like we do for the rest of stuff.
> I think this will make what Zachary is proposing easier to do, and it
> will make the test writing less awkard.
> 
> 
> What do you think? I'm ready to chip in on this if we agree to go down
> this way...
> 
> pl
> 
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> 
> 
> 
> -- 
> -Todd
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev