[lldb-dev] Test suite rebuilding test executables many times

[Zachary Turner via lldb-dev]

I think I'm goign to abandon this idea for now.  Mostly because I've found
a workaround which is a) one line of code and b) only affects windows.  So
the impact of this workaround is narrower, no risk of messing up any tests
which depend on cleaning, and gives more time to come up with a more
comprehensive solution, such as something like I and Pavel proposed earlier.

On Wed, Aug 26, 2015 at 10:24 AM Todd Fiala <todd.fiala at gmail.com> wrote:

> Not sure if this is relevant, but I seem to recall the remote test
> execution would spin off each test method run (test case level, not test
> suite level) into a new directory.  I don't think that would be inherently
> broken by a no-clean scenario but we'd want to make sure it doesn't break.
>
> -Todd
>
> On Wed, Aug 26, 2015 at 7:52 AM, Zachary Turner via lldb-dev <
> lldb-dev at lists.llvm.org> wrote:
>
>> I thought of this too and I started prototyping it.
>>
>> The issue that I ran into is that dsym and dwarf tests can all be
>> xfailed, skipped, etc for different reasons, so if there is one method
>> body, you need a way to still define the set of conditions under which dsym
>> and dwarf tests should run, skip, xfail, timeout, etc.
>>
>> Do you want to start writing
>> @skipIfDwarfAndOsIsLinuxButCompilerIsNotClang?  Because I know I don't
>> want to deal with the combinatorial explosion of decorators that would
>> result :)
>>
>> I have some ideas here as well, for example I think we only actually need
>> 1 decorator that we can configure via keyword arguments that can handle
>> arbitrarily complex scenarios of xfailing, debug infos, etc.  e.g.
>> @lldb_test(debug_types="dwo,dwarf", xfail = {...}, skip = {...}).  But
>> it's all unrelated to the original problem I'm trying to solve.  So I think
>> it would be good to design a solution for that, but to do it separately.
>>
>> The nice thing about just changing the default from clean to not clean is
>> that it's about a 3 line change, has potentially large speed improvements
>> across the board, and also fixes bugs.
>>
>>
>>
>> On Wed, Aug 26, 2015 at 2:01 AM Pavel Labath <labath at google.com> wrote:
>>
>>> On 26 August 2015 at 06:14, Zachary Turner via lldb-dev
>>> <lldb-dev at lists.llvm.org> wrote:
>>> >
>>> > I'll wait and see if anyone can remember which tests rebuild binaries
>>> on
>>> > purpose.  Otherwise I will try to look through them and see if I can
>>> figure
>>> > it out.
>>>
>>> TestInferiorChanged is one that I remember.
>>>
>>> I think this is a good thing to do, but it will need to be done with a
>>> steady hand.
>>>
>>>
>>>
>>> I was also thinking about the dsym/dwo tests.. Instead of basically
>>> having a copy of each test for dwarf and dsym (and soon also dwo), how
>>> about having just one test, and have some higher level logic (the test
>>> runner) know that it needs to execute each test multiple times. The
>>> tests would then just do a buildDefault() (or something) and on the
>>> first run it would build normal dwarf, on the second one dsym, etc. If
>>> we need to run a test only for some combination of debug infos, we
>>> could have @skipIfDsym annotations, like we do for the rest of stuff.
>>> I think this will make what Zachary is proposing easier to do, and it
>>> will make the test writing less awkard.
>>>
>>>
>>> What do you think? I'm ready to chip in on this if we agree to go down
>>> this way...
>>>
>>> pl
>>>
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
>
> --
> -Todd
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150826/a042839b/attachment.html>