[lldb-dev] possible test runner speedup
Todd Fiala via lldb-dev
lldb-dev at lists.llvm.org
Mon Sep 28 09:15:27 PDT 2015
On Mon, Sep 28, 2015 at 2:12 AM, Pavel Labath <labath at google.com> wrote:
> Interesting results. We were discussing the same thing last week. I
> was somewhat skeptical to the ideal as I am afraid of increased
> flakyness -- LLDB has hardcoded timeout values in a lot of places, and
> with increased cpu contention, we might start to see this code failing
> because the other side was slower than expected.
Yeah, when I first did this a while back (just the parallel piece, not the
speedup of it), I think it uncovered a ton of races/flakiness. Much of
that has been addressed, but nothing like forcing more contention to work
out more of them.
> I have now tried running the test suite with a higher number of
> threads and it seems it working quite alright, so I think we can try
> that. I'll watch the buildbots for signs of increased flakyness. FWIW,
> it did not speed up the tests for me at all, but I guess that's
> expected, as it ran in 90 seconds even before that, and the limiting
> factor there probably is the longest test.
Yeah, all those lengthy debugserver/lldb-server tests I wrote a while back
tend to have a long tail, and I'm sure you folks have only extended them
since then. I have some thoughts on improving the job scheduling in the
future (finer grain) but not in the short term.
> As for the timeout, I have set the value to 4 minutes, based on the
> formula: 2 * (number of seconds when I first started noticing flaky
> tests in the slowest configuration). The slowest configuration here
> was running the test suite on a debug build of lldb and using debug
> ToT clang to compile the inferiors (believe it or not, this makes a
> big difference). So there is still some leeway here, but if you're
> gonna reduce the default, please make sure it is enough for the
> slowest test configuration also. For faster configurations you can
> always override the timeout manually.
Okay. I'm not planning on touching the timeout values at this point, just
noting that they may be worth messing with in the future.
I'd like to add a very lightweight stats database (that's already too heavy
a word, but conveys the idea) that can be arranged to persist across builds
(that last part specifically for buildbots). With just a tiny bit of
historical data when available, job scheduling to reduce total time of test
run could be improved. And with some finer grain scheduling which I plan
to look into mid-term via a mechanism to prevent a single test method
hangup from knocking out the rest of the test methods in the same run, we
could do even better yet. But I'll bring this up when I get closer to
having some time to work on it. (And that's not yet!)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lldb-dev