[llvm-dev] Build status expectations for experimental targets

[Mehdi Amini via llvm-dev]

> On Feb 3, 2017, at 4:18 AM, Tobias Grosser via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> On Fri, Feb 3, 2017, at 11:37 AM, Dylan McKay via llvm-dev wrote:
>> Hey all,
>> 
>> Every few weeks, a change is committed to trunk that breaks the AVR
>> buildbot.
>> 
>> A problem presents when commit authors do not fix the build, and just
>> leave
>> it because it passes on the core buildbots. The build stays red for a few
>> days until I go and check it. In the meantime, it likely causes spam for
>> most if not all developers when they commit new code.
>> 
>> All commits should keep master green, but what is the expectation for
>> experimental backends? Is it reasonable to expect all developers who
>> commit
>> code to ensure tests pass on the AVR backend?
>> 
>> On top of this, is there any way to notify maintainers of a backend when
>> a
>> buildbot has been failing for some time? I imagine other experimental
>> backends have run into the same problems.
> 
> Hi Dylan,
> 
> this probably depends on what we want experimental targets to be and how
> they are different from normal targets. From my developer perspective,
> anything that is not enabled by default and is not checked by default in
> a normal LLVM build is something a normal developer should not need to
> worry about. As such, I do not
> expect committers to investigate bugs triggered in the AVR backend. If
> we would believe the AVR backend is stable enough, such that users can
> rely on it and such that other developers are unlikely to trigger bugs
> in the AVR backend, the AVR backend should most likely be promoted to a
> stable backend.
> 
> As a result of this, I would also expect buildbots of the AVR backend to
> not send any emails to the general public, but to instead send emails to
> the buildbot owner and maintainer of the AVR backend.
> 
> In my perspective, it is the job of the buildbot owner of an
> experimental backend to investigate failures in this backend and to
> clarify if a failure is caused by a bug in a recent change or rather by
> a bug in the AVR backend.
> If it seems as if the original change committed is incorrect, the
> buildbot owner should work with the committer of the corresponding patch
> to get the bug resolved.
> 
> Obviously, the answer I gave above is very black and white. In reality,
> the real answer depends on the maturity of the backend and the quality
> of the availability of the buildbot owner. What I wrote above has a
> rather new and unstable backend in mind -- especially the category of
> backend we would expect to become experimental in the first place. The
> more stable a backend is, the more rare breakages are, the faster bugs
> are fixed, and the more likely it is that breakages are due to bugs in
> LLVM commits, the more it makes sense to have a buildbot that sends out
> emails to the actual committers.
> 
> In Polly I have two kinds of buildbots. A set of buildbots which are
> experimental and do not send emails ever and a set of bots for which I
> know they have a very low rate of failures (every couple of weeks) and
> for which I also make sure that I myself can either fix or XFAIL any
> issues very quickly (commonly at most 2-3 failing builds during working
> hours). I do not expect anybody to fix failures in polly. However, I
> made the experience that people are kind enough to help fixing problems,
> if the bot generally has a reputation of being green most of the time.
> 
> In your case I suggest to make the buildbot experimental, disable emails
> to committers and add yourself as email receiver. When you can make sure
> the bot sends very rarely emails and you yourself can make sure bugs are
> addressed / XFAILED within a very small delay and this has been proven
> to work for a couple of weeks, you could carefully consider of enabling
> emails to other again.

If I was in this position, I’d also configure the bot to build *only* the AVR backend. That’s help make sure that an email does get send when a test fails in the X86 backend.

Best,

— 
Mehdi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170203/f08c46c6/attachment.html>