[PATCH] D114325: Add a best practice section on how to configure a fast builder

[David Blaikie via Phabricator via llvm-commits]

dblaikie added a comment.

yeah, generally I'm OK with all of this - eventually figuring out ways to migrate existing builders that are way out-of-spec (hours of blame list, etc) out of blame email notifications, back to staging or the like. (& yeah, as @mehdi_amini said, seems totally suitable that staging bots should (I think even by default, really) email builder owners on failure (possibly on every failure, not only green to red transitions))



================
Comment at: llvm/docs/HowToAddABuilder.rst:154-155
+
+As mentioned above, we generally have a strong preference for
+builders which can build every commit as they come in.  This section
+includes best practices and some recommendations as to how to achieve
----------------
reames wrote:
> reames wrote:
> > rengolin wrote:
> > > dblaikie wrote:
> > > > do we have any builders that achieve this consistently (I wouldn't think so, given the resources required)? Maybe worth rephrasing if it's  not actually achievable/achieved generally to something more in line with the practical reality?
> > > > 
> > > > If this document is more aspirational/trying to set a fairly new (albeit good, but perhaps not feasible?) direction - maybe it'd be more suitable in a different form/forum?
> > > I don't think we have many, if any, but I interpreted it as "preference" and "best practices", not that we don't accept others. I agree we shouldn't be discouraging people to set buildbots if they can't follow these guidelines.
> > As noted in the top level comment, we have a bunch of builders which do keep up building every commit.  
> > 
> > This is aspirational, but only in the sense that a new builder which can't meet this bar has to explain why it's still worthwhile having as a notifying builder.  We may accept it, but the burden of justification is definitely on the bot owner.  
> > 
> > The main glide path I see - which we need better infrastructure for - is allowing "small" (2-3) commit batches as a graceful fallback when fully keeping up isn't practical.  
> Ok, let me walk my last comment back a bit.
> 
> As written, this is a best practice.  It *does not* require any higher burden of justification for new bots, and there's nothing in the document which currently frames it that way.  (Or at least, if there is, the wording is a mistake and I'll fix that.)
> 
> Longer term, I *do* think we should be rejecting build bots which can't demonstrate a worthwhile tradeoff between benefit to single config and community impact.  However, that needs a lot of broader discussion first.   IMO, we have several today which do not justify their existence.
Fair enough.

I like the idea, I'm just not sure how feasible it is - given the number of buildbots we have that don't meet that bar today. How much testing we would lose if we made it a requirement (versus how much better test coverage we would gain from folks able to, but unaware of/unable to justify the extra hardware without that sort of encouragement/pushback)

I guess what I might say is "we generally have a strong preference for builders which can build every commit as they come in" - I'm not sure that the community behavior/norms/buildbot configurations so far indicate that "we" have that "strong preference". I think it's a good thing to have but might be an overreach to claim we do have it without some broader consensus gathering?

But happy to step back and leave you to this/leave it as-is too.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D114325/new/

https://reviews.llvm.org/D114325