[lldb-dev] possible test runner speedup

[Todd Fiala via lldb-dev]

On Mon, Sep 28, 2015 at 2:12 AM, Pavel Labath <labath at google.com> wrote:

> Hi,
>
> Interesting results. We were discussing the same thing last week. I
> was somewhat skeptical to the ideal as I am afraid of increased
> flakyness -- LLDB has hardcoded timeout values in a lot of places, and
> with increased cpu contention, we might start to see this code failing
> because the other side was slower than expected.
>

Yeah, when I first did this a while back (just the parallel piece, not the
speedup of it), I think it uncovered a ton of races/flakiness.  Much of
that has been addressed, but nothing like forcing more contention to work
out more of them.


>
> I have now tried running the test suite with a higher number of
> threads and it seems it working quite alright, so I think we can try
> that. I'll watch the buildbots for signs of increased flakyness. FWIW,
> it did not speed up the tests for me at all, but I guess that's
> expected, as it ran in 90 seconds even before that, and the limiting
> factor there probably is the longest test.
>
>
Yeah, all those lengthy debugserver/lldb-server tests I wrote a while back
tend to have a long tail, and I'm sure you folks have only extended them
since then.  I have some thoughts on improving the job scheduling in the
future (finer grain) but not in the short term.


> As for the timeout, I have set the value to 4 minutes, based on the
> formula: 2 * (number of seconds when I first started noticing flaky
> tests in the slowest configuration). The slowest configuration here
> was running the test suite on a debug build of lldb and using debug
> ToT clang to compile the inferiors (believe it or not, this makes a
> big difference). So there is still some leeway here, but if you're
> gonna reduce the default, please make sure it is enough for the
> slowest test configuration also. For faster configurations you can
> always override the timeout manually.
>
> pl
>

Okay.  I'm not planning on touching the timeout values at this point, just
noting that they may be worth messing with in the future.

I'd like to add a very lightweight stats database (that's already too heavy
a word, but conveys the idea) that can be arranged to persist across builds
(that last part specifically for buildbots).  With just a tiny bit of
historical data when available, job scheduling to reduce total time of test
run could be improved.  And with some finer grain scheduling which I plan
to look into mid-term via a mechanism to prevent a single test method
hangup from knocking out the rest of the test methods in the same run, we
could do even better yet.  But I'll bring this up when I get closer to
having some time to work on it.  (And that's not yet!)

-- 
-Todd
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20150928/1cdd967d/attachment.html>