[llvm-dev] False positive notifications around commit notifications

[Florian Hahn via llvm-dev]

Hi Philip,

> On Sep 9, 2021, at 23:18, Philip Reames via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> Flaky Builders
> 
> ex: https://lab.llvm.org/buildbot/#/builders/68/builds/18250 <https://lab.llvm.org/buildbot/#/builders/68/builds/18250>
> We have many build bots which are not entirely stable.  It's gotten to the point where I *expect* failure notifications on literally every change I land.  I've been trying to reach out to individual build bot owners to get issues resolved, and to their credit, most owners have been very responsive.  However, we have enough builders that the situation isn't getting meaningful better.
> 
> Recommendation: Introduce specific "test commits" whose only purpose is to run the CI infrastructure.  Any builder which notifies of failure on such a commit (and only said commit) is disabled without discussion until human action is taken by the bot owner to re-enable.  The idea here is to a) automate the process, and b) shift the responsibility of action to the bot owner for any flaky bot.  
Thanks for raising this issue! My experience matches what you are describing. The false positive rate for me is seems to be at least 10 false positives due to flakiness to 1 real failure. 

I think it would be good to have some sort of policy spelling out the requirements for having notification enabled for a buildbot, with a process that makes it easy to disable flaky bots until the owners can make them more stable. It would be good if notifications could be disabled without requiring contacting/interventions from individual owners, but I am not sure if that’s possible with buildbot.

Cheers,
Florian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210922/110ffcf7/attachment.html>