[llvm-dev] LLVM Releases: Upstream vs. Downstream / Distros

[Jeff Law via llvm-dev]

On 05/11/2016 08:08 AM, Renato Golin wrote:
>
> Shall we sync our upstream release with the bulk of other downstream
> ones as well as OS distributions?
I suspect that'll, in general, be hard.

The downstream consumers are all going to have their own needs & 
schedules and as a result I doubt you'll find any upstream release 
schedule & cadence that is acceptable to enough downstream consumers to 
make this viable.

With that in mind, I'm a proponent of timing based schedules with as 
much predictability as can be made.  That allows the downstream 
consumers to make informed decisions based on likely release dates.


>
>
> This work involves a *lot* of premises that are not encoded yet, so
> we'll need a lot of work from all of us. But from the recent problems
> with GCC abi_tag and the arduous job of downstream release managers to
> know which patches to pick, I think there has been a lot of wasted
> effort by everyone, and that generates stress, conflicts, etc.
Ideally as we continue to open more lines of communication we won't run 
into anything as problematical as the abi_tag stuff.  While it's 
important, I wouldn't make it the primary driver for where you're trying 
to go.

WRT downstream consumers.  The more the downstream consumer is wired 
into the development community, the more risk (via patches) the 
downstream consumer can reasonably take.  A downstream consumer without 
intimate knowledge of the issues probably shouldn't be taking 
incomplete/unapproved patches and applying them to their tree.


>
>   1. Timing
>
> Many downstream release managers, as well as distro maintainers have
> complained about the timing of our releases, and how unreliable they
> are, and how that makes it hard for them to plan their own branches,
> cherry-picks and merges. If we release too early, they miss out
> important optimisations, if we do too late, they'll have to branch
> "just before" and risk having to back-port late fixes to their own
> modified trees.
And this just gets bigger as the project gets more downstream consumers. 
    Thus I think you pick a time based release schedule, whatever it may 
be and the downstream consumers can then adjust.

Note that this can have the effect of encouraging them to engage more 
upstream to ensure issues of concern to them are addressed in a timely 
manner.

>
>   2. Process
>
> Our release process is *very* lean, and that's what makes it
> quasi-chaotic. In the beginning, not many people / companies wanted to
> help or cared about the releases, so the process was what whomever was
> doing, did. The major release process is now better defined, but the
> same happened to the minor releases.
>
> For example, we have no defined date to start, or to end. We have no
> assigned people to do the official releases, or test the supported
> targets. We still rely on voluntary work from all parties. That's ok
> when the release is just "a point in time", but if downstream releases
> and OS distributions start relying on our releases, we really should
> get a bit more professional.
Can't argue with getting a bit more structured, but watch out for going 
too far.  I'd really like to squish down the release phase on the GCC 
side, but it's damn hard at this point.


>
> A few (random) ideas:
>
>  * We should have predictable release times, both for starting it and
> finishing it. There will be complications, but we should treat them as
> the exception, not the rule.
Yes.

>  * We should have appointed members of the community that would be
> responsible for those releases, in the same way we have code owners
> (volunteers, but no less responsible), so that we can guarantee a
> consistent validation across all relevant targets. This goes beyond
> x86/ARM/MIPS/PPC and includes the other targets like AMD, NVidia, BPF,
> etc.
Good luck ;-)  Don't take this wrong, but wrangling volunteers into 
release work is hard.  It's sometimes hard to find a way to motivate 
them to focus on issues important for the release when there's new 
development work they want to be doing.

Mark Mitchell found one good tool for that in his years as the GCC 
release manager -- namely tightening what was allowed on the trunk as 
the desired release date got closer.  ie, there's a free-for-all period, 
then just bugfixes, then just regression fixes, then just doc fixes. 
Developers then had a clear vested interest in moving the release 
forward -- they couldn't commit their new development work until the 
release manager opened the trunk for new development.


>  * OS distribution managers should test on their builds, too. I know
> FreeBSD and Mandriva build by default with Clang. I know that Debian
> has an experimental build. I know that RedHat and Ubuntu have LLVM
> packages that they do care. All that has to be tested *at least* every
> major release, but hopefully on all releases. (those who already do
> that, thank you!)
LLVM's usage on the Fedora side is still small, and smaller still within 
Red Hat.  But we do have an interest in this stuff "just working". 
Given current staffing levels I would expect Fedora to follow the 
upstream releases closely with minimal changes.

>  * Every *new* bug found in any of those downstream tests should be
> reported in Bugzilla with the appropriate category (critical / major /
> minor). All major bugs have to be closed for the release to be out,
> etc. (the specific process will have to be agreed and documented).
Yes, GCC has a similar policy and it has worked reasonably well.  In 
fact it's a good lever for the release manager if you've got a locked 
trunk.  If the developers don't address the issues, then the release 
doesn't branch and the trunk doesn't open for development.  It aligns 
the release manager and a good chunk of the development team's goals.


jeff