[LLVMdev] [cfe-dev] [3.6 Release] RC3 has been tagged

[Tom Honermann]

On 02/18/2015 04:12 PM, Renato Golin wrote:
> On 18 February 2015 at 20:59, Tom Honermann <thonermann at coverity.com> wrote:
>> My company also does not use ToT.  Being able to associate a product with a
>> well-known release is *very* important to us.  It enables communication with
>> our customers.  It allows us to determine compatibility between our Clang
>> derived product and other Clang derived products.  It allows us to easily
>> discuss groups of feature sets.
>
> It's good to know that more and more people are actually interested in
> the releases. Am I right to assume that you test the release
> candidates as we put them forward with your own products? It'd be good
> to get that extra level of testing during releases.

Unfortunately, we don't currently have the resources to do so.  When we 
upgrade, we upgrade to the most recent release and perform testing on 
top of it.  We call Clang APIs directly, so upgrades are always a bit 
costly as we adjust to changing APIs (btw, that isn't a complaint.  We 
appreciate the need for APIs to evolve).

> After all, a release is only as good as the people testing it. If it
> works for you, it's good for you, and so on. If there are enough
> people testing and reporting bugs, we'll end up with a very stable
> product, even if we don't realise at a first glance.

That is true, of course.  The Clang community clearly takes testing 
seriously.  I appreciate that I see push back on patch submissions that 
do not include a test case (even if that means some of the patches I've 
submitted have not been accepted because I haven't (yet) found the time 
to code a proper test).  From my perspective, passing the test suite is 
a good indicator of release quality.

>> I personally find it rather crazy that developers are expected to use ToT
>> (and then determine stability on their own).  Why have an RC series at all
>> if that is the case?
>
> Core developers tend to use ToT. External users tend to use stable
> commits (you can check the already existing buildbots for that) or
> releases + cherry-pick. No one I know uses pure releases and expect
> them to be perfect.

We use pure releases and cherry-pick later commits for problems we 
encounter or see discussed on the mailing list that we think might 
impact us.  The performance issue being discussed is one we would want 
to cherry-pick a fix for if/when we upgrade to a release that has the 
regression (assuming I remember this discussion if/when we do so). 
Having such issues fixed prior to the release would be a benefit to us. 
  We don't expect the releases to be perfect.

> The RC series is to fix the urgent bugs so we can call the release
> minimally stable. Right now it is, and the only serious problem I know
> is the multiple regressions in performance in that phoronix run. I
> can't vouch for those numbers and not many people cared about that to
> make it a blocker, so all in all, the release is "good".

Understood.

> However, it's not perfect, and that's why we started doing
> dot-releases two years ago and why I still think it's worth.

I agree those are worthwhile.  We do monitor those.

> I can only see two solutions to external users that rely on releases
> being always better than the previous:
>
> 1. Test the full release, report all problems, wait for dot-release 1, re-base.
>
> 2. Be an active part in the release process, fix bugs (not just report
> it), keeping in mind that we normally don't spend more than a month
> during a release cycle.

Agreed.  Unfortunately, we are resource constrained (as is everyone) and 
can only do so much.

Tom.