[llvm-dev] Clarification on expectations of buildbot email notifications

Wed Feb 20 02:11:05 PST 2019

On Tue, Feb 19, 2019 at 1:29 PM Reid Kleckner via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> I don't think whether a buildbot sends email should have anything to do
> with whether we revert to green or not. Very often, developers commit
> patches that cause regressions not caught by our buildbots. If the
> regression is severe enough, then I think community members have the right,
> and perhaps responsibility, to revert the change that caused it. Our team
> maintains bots that build chrome with trunk versions of clang, and we
> identify many regressions this way and end up doing many reverts as a
> result. I think it's important to continue this practice so that we don't
> let multiple regressions pile up.
>
> I think what's important, and what we should, after this discussion
> concludes, put in the developer policy, is that the person doing the revert
> has the responsibility to do their best to help the patch author reproduce
> the problem or at least understand the bug.
>
> This can take many forms. They can link directly to an LLVM buildbot,
> which should be self-explanatory as far as reproduction goes. It can be an
> unreduced crash report. If they're nice, they can use CReduce to make it
> smaller. But, a reverter can't just say "Revert rNNN, breaks
> $RANDOM_PROJECT on x86_64-linux-gu". If they add, "reduction forthcoming"
> and they deliver on that promise, I think we should support that.
>
> In other words, the bar to revert should be low, so we can do it fast and
> save downstream consumers time and effort. If someone isn't making a good
> faith effort to follow up after a revert, then authors have a right to push
> back.
>

I really strongly endorse this approach. This, IMO, is the crux of
revert-to-green: somewhat regardless of the source of green vs. red, we
need to revert quickly and with relatively low bar. The result of a revert
is a shared obligation between reverter and author to find a path forward
and Reid nicely outlines how the reverter can address their end of the
bargain.

I want to emphasize that "quickly" here often (but definitely not always)
needs to be much shorter than "a few days" or even "a day" due to the rate
of incoming patches and the need to minimize compound failures hiding
precise regression signal.

Anyways, +1 =]
-Chandler

>
> I agree with Paul that we should remove the text about checking nightly
> builders. That suggestion seems a bit dated.
>
> On Tue, Feb 19, 2019 at 11:22 AM Zachary Turner via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi all,
>>
>> Over the past year or so, all of us have broken the buildbots on many
>> occasions.  Usually we get notified on IRC, or via an buildbot email
>> notification sent to everyone on the blamelist.
>> If I happen to be on IRC I'll see the notification, but if not, the next
>> best thing is an email that was automatically sent to me (along with
>> everyone else on the blamelist) from the buildbot with information about
>> the failure.
>> And then finally, I'll occasionally get a response to my commit message
>> telling me that it's broken, and the patch may be reverted with information
>> in the commit message explaining which bot was broken and providing a link
>> to it.
>>
>> However, we have some buildbots on the public waterfall which are
>> specifically configured not to send emails to people.  In some cases it's
>> because the bots are experimental, but there are a handful where the
>> reasoning I've been given is that it "wastes peoples time and contributes
>> to build blindness", but we are still expected to keep them green (usually
>> by people manually reaching out to us when they fail, or patches getting
>> reverted and us getting notified of the revert).
>>
>> It is this last case that I'm concerned about, as it appears to be in
>> direct conflict with our own developer policy [
>> https://llvm.org/docs/DeveloperPolicy.html#id14], which states this
>> -----
>> We prefer for this to be handled before submission but understand that it
>> isn’t possible to test all of this for every submission. Our build bots and
>> nightly testing infrastructure normally finds these problems. A good rule
>> of thumb is to check the nightly testers for regressions the day after your
>> change. Build bots will directly email you if a group of commits that
>> included yours caused a failure. You are expected to check the build bot
>> messages to see if they are your fault and, if so, fix the breakage.
>>
>> Commits that violate these quality standards (e.g. are very broken) may
>> be reverted. This is necessary when the change blocks other developers from
>> making progress. The developer is welcome to re-commit the change after the
>> problem has been fixed.
>>
>> -----
>>
>> I'm sending this email to get a sense of the community's views on this
>> matter.  If I'm correctly reading between the lines in the above passage,
>> buildbots which do not send emails should not be subject to the
>> revert-to-green policy.  To be honest, it's actually not even clear from
>> reading the above passage where the burden of fixing a "broken" patch on a
>> silent buildbot lies at all - with the patch author or with the bot
>> maintainer.
>>
>>
>> Would anyone care to weigh in with an unbiased opinion here?
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190220/e9a5861d/attachment.html>