[LLVMdev] Buildbots: Apology and Explanation

[Bill Wendling]

I need to apologize to many of you, especially Gabor, for my vitriol  
over buildbot breakages. I get very enraged by breakages that occur  
over long periods of time. Here's an attempt to explain why I fly off  
the handle at these.

Before we had buildbots, a random patch would break the system. It  
would sometimes take a whole day to determine which patch broke it. I  
came up with the idea of using builbots. We started using them  
internally at Apple (unfortunately, those machines are still  
unavailable to non-Apple employees). It quickly spread so that some  
were set up at Google. It was a huge win for us.

However, it's only a huge win if people pay attention to the error  
reports. People didn't. At first, there were a lot of "false  
negatives". Those were fixed. Then the reporting scheme wasn't great.  
Those were also fixed. Still, people would break the builds and let  
things go for hours or days at a time. This feels like a slap in the  
face to me. After so much hard work, people are ignoring the one piece  
of quality assurance that we are afforded in the LLVM community. In  
the meantime, many many patches are going in which aren't being  
tested. If something breaks after the initial breakage, then it's that  
much harder to determine what's going on. It's a lot of wasted effort.

Couple that with the fact that we at Apple (and other place, to be  
sure) rely upon ToT to be fairly stable so that we can create branches  
from ToT at a moment's notice, and you get a really bad and stressful  
situation.

So, yes, I take it personally when breakages are ignored. It's an  
unfortunate part of my personality that I'm trying to work on, but on  
occasion gets the better of me, and I lash out.

Again, my apologies. Please watch what the buildbots have to say. Even  
if you can't see the log file message, try to see if there's something  
you might have missed. Someone with access to the Apple-only machines  
would be happy to forward the error message to you if you think your  
patch might be the cause.

Sincerely,
Bill (Monkeypox)