[lldb-dev] serialized, low-load test pass in parallel test runner

[Todd Fiala via lldb-dev]

On Fri, Nov 27, 2015 at 12:34 PM, Reid Kleckner <rnk at google.com> wrote:

> Chromium's test framework uses the same technique. It has the potential to
> really slow things down if you have a lot of failing tests. You might want
> some kind of threshold for giving up, I.e. here's 50 failures, I'll stop
> running the rest so devs see
>
results sooner.
>
> Otherwise, yeah, this seems reasonable for lldb.
>
>
Thanks, Reid!  The max failure threshold seems like a good idea if/when I
put in an auto re-run-under-low-load mechanism.

I think I will start with that (the auto mode + threshold) to get it up and
running, then add the test markers and the required opt-in mode that we'll
default with per Pavel's comments.

-Todd

>

> Sent from phone
> On Nov 27, 2015 10:57 AM, "Todd Fiala via lldb-dev" <
> lldb-dev at lists.llvm.org> wrote:
>
>> Hi all,
>>
>> On OS X (and frankly on Linux sometimes as well, but predominently OS X),
>> we have tests that will sometimes fail when under significant load (e.g.
>> running the concurrent test suite, exacerbated if we crank up the number of
>> threads, but bad enough if we run at "number of concurrent workers ==
>> number of logical cores").
>>
>> I'm planning on adding a serialized, one-worker-only phase to the end of
>> the concurrent test run, where the load is much lighter since only one
>> worker will be processing at that phase.  Then, for tests that fail in the
>> first run, I'd re-run them in the serialized, single worker test run
>> phase.  On the OS X side, this would eliminate a significant number of test
>> failures that are both hard to diagnose and hard to justify spending
>> significant amounts of time on in the short run.  (There's a whole other
>> conversation to have about fixing them for real, i.e. working through all
>> the race and/or faulty test logic assumptions that are stressed to the max
>> under heavier load, but practically speaking, there are so many of them
>> that this is going to be impractical to address in the short/mid term.).
>>
>> My question to all of you is if we'd want this functionality in top of
>> tree llvm.org lldb.  If not, I'll do it in one of our branches.  If so,
>> we can talk about possibly having a category or some other mechanism if we
>> want to mark those tests that are eligible to be run in the follow-up
>> serialized, low-load pass.  Up front I was just going to allow any test to
>> fall into that bucket.  The one benefit to having it in top of tree
>> llvm.org is that, once I enable test reporting on the green dragon
>> public llvm.org OS X LLDB builder, that builder will be able to take
>> advantage of this, and will most certainly tag fewer changes as breaking a
>> test (in the case where the test is just one of the many that fail under
>> high load).
>>
>> Let me know your thoughts either way.
>>
>> Thanks!
>> --
>> -Todd
>>
>> _______________________________________________
>> lldb-dev mailing list
>> lldb-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>


-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151128/24ca876c/attachment-0001.html>