[llvm-dev] [cfe-dev] Buildbot Noise

[Renato Golin via llvm-dev]

On 16 October 2015 at 15:17, Robinson, Paul
<Paul_Robinson at playstation.sony.com> wrote:
> But if
> there are new fails, the blame mailer can do a set-difference and report
> only the new ones. That would reduce the noise a bit, hmm?

Hi Paul,

The danger there is that it'd be easier to "get used" to having some
failures as long as you don't have "new" failures. Every place I
worked that supported that philosophy, ended up with all bots
"orange". It's never the intention, but it's almost always the
inevitable consequence. In a small team, or a single company, it may
be a lot easier to move them back to green, but in an open community,
it's not that easy, nor that quick.

The way we work with the same concept, as David mentioned repeatedly,
is to use XFAILs. It is essentially the same thing, except that "it
hurts more" to mark an XFAIL than to see a different shade of red, so
we're more reluctant to ignore them.

Plus, an orange bot that becomes red (new failures) will itself become
orange as time passes, or new failures show up. If we end up with that
many shades of red, understanding the difference will become harder,
and the value will decrease.

cheers,
--renato