[llvm-dev] Wanted: a way to test changes before breaking all the build bots.

[Renato Golin via llvm-dev]

On 12 April 2016 at 16:13, James Y Knight via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> I'd love to have some reasonable way that I could try out a commit *other
> than* submitting it and hoping for the best, so that sort of thing can be
> avoided.

Hi James,

The proposal for pre-commit buildbots is a long lived one...

The accepted reality (not a consensus), is that it's more
cost-effective to spend time and money on real builders for new
configurations than duplicating builders for pre-commit. Given that we
have a lax policy on reverting patches anyway, people feel comfortable
in breaking a few things here and there.

Personally, I would love some pre-commit buildbot. I really dislike
the series of fix-up patches that come after breaking bots, making
them impossible to revert.

I'm assuming that the number of pre-commit patches are larger than
commit ones, given that people will be trying them out as they bake.
Therefore, such a pre-commit infrastructure would have to be bigger,
per configuration, than our current buildbot infrastructure. If we
have one builder per architecture, but multiple slaves, that could be
scalable, but they wouldn't catch all bugs (which is ok).

We'd also want all those pre-commit builders to be monitoring the same
branch across all projects. And that's where testing patches that
cross project boundaries become complicated. LLVM has around 12 major
repositories, that get built and tested regularly, and the algorithm
that would pick a branch and test in Buildbot is not trivial.

The alternative would be Jenkins, uploading a patch (that could
potentially span across multiple repositories), and then test. The
infrastructure would be *much* simpler, but we'd have to create a
parallel infrastructure, not necessarily synchronised with the main
one. Given that Apple is moving to Jenkins internally, I don't see
this as a bad thing, at least not for a completely separated
infrastructure.

But both solutions will have the volume issue. You may have to wait
longer to have your patch validated on a tree that might be, now,
outdated, than to commit, break & fix. This is a somewhat contentious
issue in this community.

My opinion is that we should have less haste, more quality. So,
pre-commit bots (BB or Jk or whatever) is a way forward, and I support
it. But I have been frequently outnumbered in the past... :)

cheers,
--renato