<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 9/22/21 2:45 AM, Florian Hahn wrote:<br>
</div>
<blockquote type="cite"
cite="mid:2EB26586-8251-42D6-B1FC-FAC4917BDC64@apple.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
Hi Philip,<br class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Sep 9, 2021, at 23:18, Philip Reames via
llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org"
class="" moz-do-not-send="true">llvm-dev@lists.llvm.org</a>>
wrote:</div>
<div class="">
<div class="">
<p class=""><b class="">Flaky Builders</b></p>
<p class="">ex: <a class="moz-txt-link-freetext"
href="https://lab.llvm.org/buildbot/#/builders/68/builds/18250"
moz-do-not-send="true">https://lab.llvm.org/buildbot/#/builders/68/builds/18250</a></p>
<p class="">We have many build bots which are not entirely
stable. It's gotten to the point where I *expect*
failure notifications on literally every change I land.
I've been trying to reach out to individual build bot
owners to get issues resolved, and to their credit, most
owners have been very responsive. However, we have
enough builders that the situation isn't getting
meaningful better.</p>
<p class="">Recommendation: Introduce specific "test
commits" whose only purpose is to run the CI
infrastructure. Any builder which notifies of failure
on such a commit (and only said commit) is disabled
without discussion until human action is taken by the
bot owner to re-enable. The idea here is to a) automate
the process, and b) shift the responsibility of action
to the bot owner for any flaky bot. <br class="">
</p>
</div>
</div>
</blockquote>
</div>
Thanks for raising this issue! My experience matches what you are
describing. The false positive rate for me is seems to be at least
10 false positives due to flakiness to 1 real failure.
<div class=""><br class="">
</div>
<div class="">I think it would be good to have some sort of policy
spelling out the requirements for having notification enabled
for a buildbot, with a process that makes it easy to disable
flaky bots until the owners can make them more stable. It would
be good if notifications could be disabled without requiring
contacting/interventions from individual owners, but I am not
sure if that’s possible with buildbot.</div>
</blockquote>
<a class="moz-txt-link-freetext" href="https://reviews.llvm.org/D112755">https://reviews.llvm.org/D112755</a> adds the first pieces of some
documented policy around build bot expectations. It does not
address the point you raise as the intent was to be a minimal
documentation of existing practice, and thus hopefully be
non-controversial, but assuming this moves forward, I plan to
revisit this topic in its own review. <br>
<blockquote type="cite"
cite="mid:2EB26586-8251-42D6-B1FC-FAC4917BDC64@apple.com">
<div class=""><br class="">
</div>
<div class="">Cheers,</div>
<div class="">Florian</div>
</blockquote>
</body>
</html>