[cfe-dev] [Openmp-dev] RFC: Release process changes

[Hans Wennborg via cfe-dev]

That all makes sense to me. The new process sgtm.

On Wed, May 27, 2020 at 5:20 AM Tom Stellard <tstellar at redhat.com> wrote:
>
> On 05/25/2020 05:48 AM, Hans Wennborg wrote:
> > On Thu, May 21, 2020 at 8:59 PM Tom Stellard via Openmp-dev
> > <openmp-dev at lists.llvm.org> wrote:
> >>
> >> Hi,
> >>
> >> I would like to propose a few changes to the LLVM release process.  The
> >> current process is documented here:  https://llvm.org/docs/HowToReleaseLLVM.html
> >>
> >> There are two parts to this proposal.  The first is a list of clarifications,
> >> which are things we are currently doing that aren't documented. The second
> >> is a list of changes which would actually modify how releases are currently
> >> managed.
> >>
> >>
> >>
> >> *** Proposed Clarifications ***
> >>
> >>
> >>
> >> **  Release manager is allowed to commit changes to the release branch without
> >>     code owner approval.  However, the release manager is encouraged to consult
> >>     with code owners or patch reviewers for non-trivial changes.
> >>
> >> It's not practical to get code owner approval every time.  Either because there
> >> is no code owner or because the number of backports is too high (e.g. pre-rc1 / pre-rc2).
> >> This proposed clarification matches how releases are currently managed.
> >
> > +1
> >
> > Maybe even stronger than "is allowed to commit", I think we should
> > really think about it as the release manager owning the branch, and
> > has full authority over what goes into it or not. Consulting code
> > owners often makes sense of course, but for many patches, consulting
> > the code owner (when there is one) is an unnecessary slowdown.
> >
> >> ** There is no official release criteria.
> >>
> >> We have time-based releases and when the release is 'ready' has been
> >> up to the discretion of the release manager.  Changing the release
> >> criteria is out of the scope of this proposal, but I do think it would
> >> be good to have a discussion about this as a community, so I'm going to
> >> start a separate thread to discuss this.
> >>
> >>
> >>
> >> *** Proposed Changes ***
> >>
> >>
> >>
> >> ** Create a time-based bug-fix release schedule.  After each major release, make
> >>    a new bug-fix release every 2 weeks for 12 weeks (6 releases total).
> >>
> >> ** Eliminate release candidates for bug-fix releases.
> >>
> >> The current unofficial bug-fix release schedule is:
> >>
> >> X.Y.1-rc1 (6 weeks after major release)
> >> X.Y.1-rc2 (10 weeks after major release)
> >> X.Y.1-final (12 weeks after major release)
> >>
> >> I think this change will improve the overall test coverage of the release branch.
> >> I don't think the branch itself or even the release candidates get the same
> >> level of testing as the final releases.  If we are consistently snapshotting
> >> the release branch and putting out releases, I think this will make it easier
> >> and thus more likely that users will test out the release branch code.
> >>
> >> Additionally, with more frequent bug-fix release it removes the need to have
> >> release candidate releases. Every bug-fix release (up until the last one)
> >> would serve the same purpose as our current release candidates in that they
> >> are intended to give users an easier way to test the code before the final
> >> release.
> >
> > My first thought is that doing all these releases sounds like a lot of
> > work. Would you be doing all of them, or would there be some other
> > arrangement? I suppose if we release this often, and also skip the
> > RCs, we might become more efficient at it :-)
> >
>
> Yes, I would plan to do all the releases.  For 9.0.1, there were
> 3 RCs, so 4 releases in total.  Doing 6 instead of 4 is not that much
> more work in my opinion.  Also, we may end up skipping releases if
> there aren't any new changes in the branch.  But doing extra
> releases would be good motivation to try to automates more parts of the
> release process.
>
> If we do feel like 6 is too many we could lengthen the interval to 3 weeks,
> which would give us just 4 releases.
>
> > Secondly, is having this many releases useful for downstream? One
> > concern might be that downstream consumers just wait for the .6 one,
> > and then there's no benefit and also no extra testing of the branch.
> > Is it mainly increasing test coverage of the branch that's the
> > motivation, or is it the demand for more bug-fix releases?
> >
>
> From me as a distro package maintainer, I'm more likely to package
> a final release than a bug-fix release.  Especially if I know there won't
> be another release candidate or final release coming very soon after.
>
> Besides increasing testing coverage, I think it helps other projects
> avoid having to do things like this: https://lkml.org/lkml/2020/5/5/1446
> In this case, the kernel release cycle is about 2 months, so they can't
> wait 3 months for a fix.
>
>
> > Not having at least one release candidate sounds a bit scary to be.
> > Without them we could get into a situation where everything works fine
> > on the release manager's machines, but is completely broken on other
> > platforms, and no way to fix until the next dot release. Maybe that
> > kind of breakage is less likely after the major release, but it still
> > seems it could make these dot releases less stable?
> >
>
> I think in order to do this we need to have a certain level of CI testing.
> I've been experimenting with this in the last few release cycles.  Right now,
> in the release branch we have tests running make-check for clang, llvm, lld,
> libclc, and running make for lldb.  Each test runs on Linux, Mac, and Windows.
> I feel comfortable enough with this that it would catch most obvious issues
> with new bug-fix releases.  Especially since the number of changes is low
> as compared to major releases.
>
> I think there would need to be some flexibility to do another bug-fix release
> ASAP if there is a major problem.
>
> -Tom
>
> >> ** Create clear rules for what kind of backports are accepted during each
> >>    release phase.
> >>
> >> * Before RC1:Patches should be limited to bug fixes, important optimization
> >>   improvements, or completion of features that were started before the branch
> >>   was created.  As with all phases, release managers and code owners can reject
> >>   patches that are deemed too invasive.
> >>
> >> * Before RC2: Patches should be limited to bug fixes or backend specific
> >>   improvements that are determined to be very safe.
> >>
> >> * Before RC3/Final: Major Release* Patches should be limited to critical
> >>   bugs or regressions.
> >>
> >> * Bug fix releases: Patches should be limited to bug fixes or very safe
> >>   and critical performance improvements.  Patches must maintain both API and
> >>   ABI compatibility with the previous major release.
> >>
> >> * Final bug fix release: Patches should be limited to critical bug fixes only.
> >
> > These sound good to me.
> >
> >
> > Thanks,
> > Hans
> >
>