[llvm-dev] Build status expectations for experimental targets

[Tobias Grosser via llvm-dev]

On Fri, Feb 3, 2017, at 11:37 AM, Dylan McKay via llvm-dev wrote:
> Hey all,
> 
> Every few weeks, a change is committed to trunk that breaks the AVR
> buildbot.
> 
> A problem presents when commit authors do not fix the build, and just
> leave
> it because it passes on the core buildbots. The build stays red for a few
> days until I go and check it. In the meantime, it likely causes spam for
> most if not all developers when they commit new code.
> 
> All commits should keep master green, but what is the expectation for
> experimental backends? Is it reasonable to expect all developers who
> commit
> code to ensure tests pass on the AVR backend?
>
> On top of this, is there any way to notify maintainers of a backend when
> a
> buildbot has been failing for some time? I imagine other experimental
> backends have run into the same problems.

Hi Dylan,

this probably depends on what we want experimental targets to be and how
they are different from normal targets. From my developer perspective,
anything that is not enabled by default and is not checked by default in
a normal LLVM build is something a normal developer should not need to
worry about. As such, I do not
expect committers to investigate bugs triggered in the AVR backend. If
we would believe the AVR backend is stable enough, such that users can
rely on it and such that other developers are unlikely to trigger bugs
in the AVR backend, the AVR backend should most likely be promoted to a
stable backend.

As a result of this, I would also expect buildbots of the AVR backend to
not send any emails to the general public, but to instead send emails to
the buildbot owner and maintainer of the AVR backend.

In my perspective, it is the job of the buildbot owner of an
experimental backend to investigate failures in this backend and to
clarify if a failure is caused by a bug in a recent change or rather by
a bug in the AVR backend.
If it seems as if the original change committed is incorrect, the
buildbot owner should work with the committer of the corresponding patch
to get the bug resolved.

Obviously, the answer I gave above is very black and white. In reality,
the real answer depends on the maturity of the backend and the quality
of the availability of the buildbot owner. What I wrote above has a
rather new and unstable backend in mind -- especially the category of
backend we would expect to become experimental in the first place. The
more stable a backend is, the more rare breakages are, the faster bugs
are fixed, and the more likely it is that breakages are due to bugs in
LLVM commits, the more it makes sense to have a buildbot that sends out
emails to the actual committers.

In Polly I have two kinds of buildbots. A set of buildbots which are
experimental and do not send emails ever and a set of bots for which I
know they have a very low rate of failures (every couple of weeks) and
for which I also make sure that I myself can either fix or XFAIL any
issues very quickly (commonly at most 2-3 failing builds during working
hours). I do not expect anybody to fix failures in polly. However, I
made the experience that people are kind enough to help fixing problems,
if the bot generally has a reputation of being green most of the time.

In your case I suggest to make the buildbot experimental, disable emails
to committers and add yourself as email receiver. When you can make sure
the bot sends very rarely emails and you yourself can make sure bugs are
addressed / XFAILED within a very small delay and this has been proven
to work for a couple of weeks, you could carefully consider of enabling
emails to other again.

Best,
Tobias