[llvm-dev] [Openmp-dev] RFC: Release qualification criteria

[Tom Stellard via llvm-dev]

On 05/25/2020 06:10 AM, Hans Wennborg wrote:
> On Thu, May 21, 2020 at 8:59 PM Tom Stellard via Openmp-dev
> <openmp-dev at lists.llvm.org> wrote:
>>
>> Hi,
>>
>> I'm splitting this discussion off of my RFC for release process
>> changes.
>>
>> We currently have no official release qualification criteria.  In
>> other words, we don't have any blocking tests that need to pass in
>> order to make a new release.
>>
>> We do time-based releases, which is not always compatible with having
>> quality-based criteria for tagging a final release.  So, I think another
>> way to look at this issue is to talk about what kinds of CI testing we
>> have for trunk and if there are any additional kinds of
>> testing (e.g. compile-time performance) that we want to prioritize.
>>
>> So, for the purposes of this discussion, I see 2 main questions:
>>
>> 1. Should we define some set of blocking tests that need to pass before a release
>>    can happen?
> 
> I suppose we could have a baseline about clang bootstrap + lit tests
> (what test-release.sh does essentially) passes on major platforms.
> 

I think for testing like this that can be easily automated we're almost
better off just adding these as CI tests from trunk rather than having
them gate releases.

-Tom

> But the really interesting question for me is really what kind of bugs
> we're considering as release blocking. It's the trade-off between
> shipping on (or not too much behind schedule) and delaying to
> investigate more issues that's tricky..
> 
>>
>> 2. What gaps do we currently have in our CI testing?
> 
> The latest release made it clear we don't track compile time very
> well, or at least not well enough to catch the regressions in that
> release early enough.
> 
> Also I think there's no 32-bit Windows buildbot anymore :/
>