[llvm-dev] Clarification on expectations of buildbot email notifications

Tue Feb 19 12:02:44 PST 2019

On 02/19/2019 11:21 AM, Zachary Turner via llvm-dev wrote:
> Hi all,
> 
> Over the past year or so, all of us have broken the buildbots on many occasions.  Usually we get notified on IRC, or via an buildbot email notification sent to everyone on the blamelist.  
> If I happen to be on IRC I'll see the notification, but if not, the next best thing is an email that was automatically sent to me (along with everyone else on the blamelist) from the buildbot with information about the failure.  
> And then finally, I'll occasionally get a response to my commit message telling me that it's broken, and the patch may be reverted with information in the commit message explaining which bot was broken and providing a link to it.
> 
> However, we have some buildbots on the public waterfall which are specifically configured not to send emails to people.  In some cases it's because the bots are experimental, but there are a handful where the reasoning I've been given is that it "wastes peoples time and contributes to build blindness", but we are still expected to keep them green (usually by people manually reaching out to us when they fail, or patches getting reverted and us getting notified of the revert).  
> 
> It is this last case that I'm concerned about, as it appears to be in direct conflict with our own developer policy [https://llvm.org/docs/DeveloperPolicy.html#id14], which states this 
> -----
> We prefer for this to be handled before submission but understand that it isn’t possible to test all of this for every submission. Our build bots and nightly testing infrastructure normally finds these problems. A good rule of thumb is to check the nightly testers for regressions the day after your change. Build bots will directly email you if a group of commits that included yours caused a failure. You are expected to check the build bot messages to see if they are your fault and, if so, fix the breakage.
> 
> Commits that violate these quality standards (e.g. are very broken) may be reverted. This is necessary when the change blocks other developers from making progress. The developer is welcome to re-commit the change after the problem has been fixed.
> 
> -----  
> 
> I'm sending this email to get a sense of the community's views on this matter.  If I'm correctly reading between the lines in the above passage, buildbots which do not send emails should not be subject to the revert-to-green policy.  To be honest, it's actually not even clear from reading the above passage where the burden of fixing a "broken" patch on a silent buildbot lies at all - with the patch author or with the bot maintainer.
> 
> 
> Would anyone care to weigh in with an unbiased opinion here?
> 

I think for any regressions whether they affect buildbots or not, the
patch author should be responsible for fixing the issue.  In my experience,
this is also usually what happens when I report a regression.

I think the responsibility of the regression reporter (which could be
a buildbot) is to provide a reasonably prompt notification of failure along
with clear instructions for reproducing the issue.  If that criteria is met,
then I think it is always fair to ask for a revert if the issue can't be fixed
in a few days.

Even without a prompt notification, though, I still think reverts are appropriate in
some cases and that the patch author should take the lead in fixing the issue.

The buildbots that automatically send notifications,though, are a little
bit of a special case, because when they are broken it affects everybody.
I think for those bots, having an immediate revert to green policy, like
we do now is fine.

I don't think it's reasonable to expect developers to monitor nightly testers,
so maybe this part of the developer policy should be changed.  There is
almost always at least one one bot that is red.

-Tom

> 
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 