[llvm-dev] False positive notifications around commit notifications

[Martin Storsjö via llvm-dev]

On Wed, 22 Sep 2021, Florian Hahn via llvm-dev wrote:

> Thanks for raising this issue! My experience matches what you are
> describing. The false positive rate for me is seems to be at least 10 false
> positives due to flakiness to 1 real failure. 
> I think it would be good to have some sort of policy spelling out the
> requirements for having notification enabled for a buildbot, with a process
> that makes it easy to disable flaky bots until the owners can make them more
> stable. It would be good if notifications could be disabled without
> requiring contacting/interventions from individual owners, but I am not sure
> if that’s possible with buildbot.

Another aspect is that some tests can be flakey - they might work 
seemingly fine in local testing but start showing up as timeouts/spurious 
failures when run in a CI/buildbot setting. And due to their flakiness, 
it's not evident when the breakage is introduced, but over time, such 
flakey tests/setups do add up, to the situation we have today.

// Martin