[llvm-dev] LLVM Releases: Upstream vs. Downstream / Distros

[Renato Golin via llvm-dev]

On 11 May 2016 at 17:16, Hans Wennborg <hans at chromium.org> wrote:
> This is a long email :-) I've made some comments inline, but I'll
> summarize my thoughts here:

Thanks Hans!

I'll respond them inline, below.


> - I think we should use the bug tracker to capture issues that affect
> releases. It would be cool if a commit hook could update bugzilla
> entries that refer to it.

That seems like a simple hook.


> At least for the major releases, I think we're doing pretty well on
> timing in terms of predictability: since 3.6, we have release every
> six months: first week of March and first week of September (+- a few
> days). Branching has been similarly predictive: mid-January and
> mid-July.

Indeed, we got a lot better more recently (last 2y), and mostly thanks
to you. :)

We used to vary 3 months +-, and now we're down to a few days.
Whatever we decide, I think we should make it official but putting it
out somewhere, so people can rely on that.

Right now, even if you're extra awesome, there's nothing telling the
distros and LLVM-based products that it will be so if someone else
takes over the responsibility, so they can't adapt.

That's what I meant by "quasi-chaotic".


> If there are many downstream releases for which shifting this schedule
> would be useful, I suppose we could do that, but it seems unlikely
> that there would be agreement on this, and changing the schedule is
> disruptive for those who depend on it.

That's the catch. If we want them to participate, the process has to
have some meaning to them. The fact that not many people do, is clear
to me what it means.

We also need to know better how many other releases already depend on
the upstream process (not just Chromium, for obvious reasons), to be
able to do an informed choice of dates and frequency.

The more well positioned and frequent we are, the more people will
help, but there's a point where the curve bends down, and the cost is
just too great. We need to find the inflection point, and that will
require some initial investigations and guesses, and a lot of fine
tuning later. But if we're all on the same page, I think we can do
that, even if it takes time.

I'm particularly concerned with Android, because they not only have
their own tree with heavily modified LLVM components (ex.
Compiler-RT), but they also build differently and so their process are
completely alien to ours. One of the key reasons why these things
happened is because:

 * They couldn't rely on our releases, as fixing bugs and back-porting
wasn't a thing back then
 * They already had their own release schedule, so aligning with ours
brought no extra benefit
 * We always expected people to work off trunk, and everyone had to
create their own process

I don't want to change how people work, just to add one more valid way
of working, which is most stable for upstream releases. :)



> The only reasonable way I see of aligning upstream releases with
> downstream schedules would be to release much more often. This works
> well in Chromium where there's a 6-week staged release schedule. This
> would mean there's always a branch going for the next release, and
> important bug fixes would get merged to that.

Full validation every 6 weeks is just not possible. But a multiple of
that, say every 3~4 months, could be much easier to work around.



> In Chromium we drive
> this from the bug tracker -- it would be very hard to scan each commit
> for things to cherry-pick. This kind of process has a high cost
> though, there has to be good infrastructure for it (buildbots on the
> branch for all targets, for example), developers have to be aware, and
> even then it's a lot of work for those doing the releases. I'm not
> sure we'd want to take this on. I'm also not sure it would be suitable
> for a compiler, where we want the releases to have long life-time.

This works because you have a closed system. As you say, Chromium is
mostly final product, not a tool to develop other products, and the
validation is a lot simpler.

With Clang, we'd want to involve external releases into it, and it
simply wouldn't scale.



> For the major releases, I've tried to do this. We could certainly
> formalize it by posting it on the web page though.

I think that'd be the first step, yes. But I wanted to start with a
good number. 2 times a year? Would 3 times improve things that much
for the outsiders? Or just moving the dates would be enough for most
people?

That's why I copied so many outsiders, so they can chime in and let us
know what would be good for *them*.


> Most importantly, those folks should get involved :-)

Indeed!


> In practice, we kind of have this for at least some of the targets.
> Maybe we should write this down somewhere instead of me asking for
> (the same) volunteers each time the release process starts?

I give consent to mark me as the ARM/AArch64 release tester for the
foreseeable future. :)

I can also help Sylvestre, Doko, Ed, Jeff, Bero etc. to test on their
system running on ARM/AArch64 hardware.

cheers,
--renato