[llvm-dev] Wanted: a way to test changes before breaking all the build bots.

[Robinson, Paul via llvm-dev]

I don't think we need perfect/complete coverage in a pre-commit
scenario, limiting it to the high-value cases would be enough.

I haven't tried to collect data but my intuition is that the most
common breakages are Windows, followed by release builds.  The latter
is something people can do themselves (although it's tedious); the
former, not so much.
--paulr

> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Renato Golin via llvm-dev
> Sent: Tuesday, April 12, 2016 8:44 AM
> To: James Y Knight
> Cc: llvm-dev
> Subject: Re: [llvm-dev] Wanted: a way to test changes before breaking all
> the build bots.
> 
> On 12 April 2016 at 16:13, James Y Knight via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> > I'd love to have some reasonable way that I could try out a commit
> *other
> > than* submitting it and hoping for the best, so that sort of thing can
> be
> > avoided.
> 
> Hi James,
> 
> The proposal for pre-commit buildbots is a long lived one...
> 
> The accepted reality (not a consensus), is that it's more
> cost-effective to spend time and money on real builders for new
> configurations than duplicating builders for pre-commit. Given that we
> have a lax policy on reverting patches anyway, people feel comfortable
> in breaking a few things here and there.
> 
> Personally, I would love some pre-commit buildbot. I really dislike
> the series of fix-up patches that come after breaking bots, making
> them impossible to revert.
> 
> I'm assuming that the number of pre-commit patches are larger than
> commit ones, given that people will be trying them out as they bake.
> Therefore, such a pre-commit infrastructure would have to be bigger,
> per configuration, than our current buildbot infrastructure. If we
> have one builder per architecture, but multiple slaves, that could be
> scalable, but they wouldn't catch all bugs (which is ok).
> 
> We'd also want all those pre-commit builders to be monitoring the same
> branch across all projects. And that's where testing patches that
> cross project boundaries become complicated. LLVM has around 12 major
> repositories, that get built and tested regularly, and the algorithm
> that would pick a branch and test in Buildbot is not trivial.
> 
> The alternative would be Jenkins, uploading a patch (that could
> potentially span across multiple repositories), and then test. The
> infrastructure would be *much* simpler, but we'd have to create a
> parallel infrastructure, not necessarily synchronised with the main
> one. Given that Apple is moving to Jenkins internally, I don't see
> this as a bad thing, at least not for a completely separated
> infrastructure.
> 
> But both solutions will have the volume issue. You may have to wait
> longer to have your patch validated on a tree that might be, now,
> outdated, than to commit, break & fix. This is a somewhat contentious
> issue in this community.
> 
> My opinion is that we should have less haste, more quality. So,
> pre-commit bots (BB or Jk or whatever) is a way forward, and I support
> it. But I have been frequently outnumbered in the past... :)
> 
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev