<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Philip,<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Sep 9, 2021, at 23:18, Philip Reames via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><div class=""><div class=""><p class=""><b class="">Flaky Builders</b></p><p class="">ex: <a class="moz-txt-link-freetext" href="https://lab.llvm.org/buildbot/#/builders/68/builds/18250">https://lab.llvm.org/buildbot/#/builders/68/builds/18250</a></p><p class="">We have many build bots which are not entirely stable. It's
gotten to the point where I *expect* failure notifications on
literally every change I land. I've been trying to reach out to
individual build bot owners to get issues resolved, and to their
credit, most owners have been very responsive. However, we have
enough builders that the situation isn't getting meaningful
better.</p><p class="">Recommendation: Introduce specific "test commits" whose only
purpose is to run the CI infrastructure. Any builder which
notifies of failure on such a commit (and only said commit) is
disabled without discussion until human action is taken by the bot
owner to re-enable. The idea here is to a) automate the process,
and b) shift the responsibility of action to the bot owner for any
flaky bot. <br class="">
</p></div></div></blockquote></div>Thanks for raising this issue! My experience matches what you are describing. The false positive rate for me is seems to be at least 10 false positives due to flakiness to 1 real failure. <div class=""><br class=""></div><div class="">I think it would be good to have some sort of policy spelling out the requirements for having notification enabled for a buildbot, with a process that makes it easy to disable flaky bots until the owners can make them more stable. It would be good if notifications could be disabled without requiring contacting/interventions from individual owners, but I am not sure if that’s possible with buildbot.</div><div class=""><br class=""></div><div class="">Cheers,</div><div class="">Florian</div></body></html>