[llvm-dev] Buildbot Noise

[Renato Golin via llvm-dev]

On 7 October 2015 at 22:44, Eric Christopher <echristo at gmail.com> wrote:
> I think this is a poor analogy. You're also ignoring the solution I gave you
> in my previous mail for slow bots.

I'm not ignoring it, I'm acting upon it. But it takes time. I don't
have infinite resources.


> If you can't give some basic stability guarantees then the bot
> is only harming the entire testing infrastructure.

Define stability. Daniel was talking about "things I can act upon".
That's so vague it means nothing. "Basic stability guarantees" is on a
similar gist.

Any universal rule you try to make will either be too lax for fast and
reliable bots, or too hard on slow and less used bots.

That's what I'm finding hard to understand. All you guys are saying is
that things are bad and need to get better. I agree completely. But
your solution is to turn off everything you don't understand or assume
it's flaky, and that's just wrong.

We had two flaky bots: Pandas and a Juno. Pandas were disabled, the
Juno was fixed. Some of our bots, however, are still slow, and we have
been asked to disable them because they were red for too long.

Most of the problem we find are bad tests from people that didn't
(obviously) test on ARM. The second most common is code that doesn't
take into account 32-bits platforms. The third most common breakages
is the sanitizer tests, which pop in and out on many platforms. The
most common long breakage is due to self-hosted Clang breaking and
making it hard to find what commit to revert or even warn the
developer.

None of those are due to instability of my buildbots. But I got
shouted at many times to disable the bot because it was "red for too
long". I find this behaviour disrespectful.

I'm now trying to get 8 more ARM boards and 3 AArch64, and I plan to
put them as redundant builders. But it takes time. Weeks to make them
work reliably, more weeks to make sure they won't fall under pressure,
more weeks to put in production and stabilise. Meanwhile, I'd
appreciate if people stopped trying to kill the others.

What else do you want us to do?

cheers,
--renato