[llvm-dev] False positive notifications around commit notifications
Philip Reames via llvm-dev
llvm-dev at lists.llvm.org
Thu Oct 28 13:56:33 PDT 2021
On 9/22/21 2:45 AM, Florian Hahn wrote:
> Hi Philip,
>> On Sep 9, 2021, at 23:18, Philip Reames via llvm-dev
>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> *Flaky Builders*
>> ex: https://lab.llvm.org/buildbot/#/builders/68/builds/18250
>> We have many build bots which are not entirely stable. It's gotten
>> to the point where I *expect* failure notifications on literally
>> every change I land. I've been trying to reach out to individual
>> build bot owners to get issues resolved, and to their credit, most
>> owners have been very responsive. However, we have enough builders
>> that the situation isn't getting meaningful better.
>> Recommendation: Introduce specific "test commits" whose only purpose
>> is to run the CI infrastructure. Any builder which notifies of
>> failure on such a commit (and only said commit) is disabled without
>> discussion until human action is taken by the bot owner to
>> re-enable. The idea here is to a) automate the process, and b) shift
>> the responsibility of action to the bot owner for any flaky bot.
> Thanks for raising this issue! My experience matches what you are
> describing. The false positive rate for me is seems to be at least 10
> false positives due to flakiness to 1 real failure.
> I think it would be good to have some sort of policy spelling out the
> requirements for having notification enabled for a buildbot, with a
> process that makes it easy to disable flaky bots until the owners can
> make them more stable. It would be good if notifications could be
> disabled without requiring contacting/interventions from individual
> owners, but I am not sure if that’s possible with buildbot.
https://reviews.llvm.org/D112755 adds the first pieces of some
documented policy around build bot expectations. It does not address
the point you raise as the intent was to be a minimal documentation of
existing practice, and thus hopefully be non-controversial, but assuming
this moves forward, I plan to revisit this topic in its own review.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the llvm-dev