[cfe-dev] [Openmp-dev] RFC: Release qualification criteria

[Hans Wennborg via cfe-dev]

On Thu, May 21, 2020 at 8:59 PM Tom Stellard via Openmp-dev
<openmp-dev at lists.llvm.org> wrote:
>
> Hi,
>
> I'm splitting this discussion off of my RFC for release process
> changes.
>
> We currently have no official release qualification criteria.  In
> other words, we don't have any blocking tests that need to pass in
> order to make a new release.
>
> We do time-based releases, which is not always compatible with having
> quality-based criteria for tagging a final release.  So, I think another
> way to look at this issue is to talk about what kinds of CI testing we
> have for trunk and if there are any additional kinds of
> testing (e.g. compile-time performance) that we want to prioritize.
>
> So, for the purposes of this discussion, I see 2 main questions:
>
> 1. Should we define some set of blocking tests that need to pass before a release
>    can happen?

I suppose we could have a baseline about clang bootstrap + lit tests
(what test-release.sh does essentially) passes on major platforms.

But the really interesting question for me is really what kind of bugs
we're considering as release blocking. It's the trade-off between
shipping on (or not too much behind schedule) and delaying to
investigate more issues that's tricky..

>
> 2. What gaps do we currently have in our CI testing?

The latest release made it clear we don't track compile time very
well, or at least not well enough to catch the regressions in that
release early enough.

Also I think there's no 32-bit Windows buildbot anymore :/