[LLVMdev] Validating LLVM

[Bill Wendling]

On Nov 12, 2008, at 3:05 PM, David Greene wrote:

> On Wednesday 12 November 2008 02:20, Bill Wendling wrote:
>> On Nov 11, 2008, at 11:00 PM, Óscar Fuentes wrote:
>>> Bill Wendling <isanbard at gmail.com> writes:
>>>
>>> [snip]
>>>
>>>> All four of the above should be run on at least a nightly basis  
>>>> (more
>>>> frequently for some, like the regression tests). Each of these are
>>>> automated, making that easy. If there are no regressions from the
>>>> above four, we could tag that revision as being potentially  
>>>> "valid".
>>>
>>> If a new test case is created (coming from a bug report or a code
>>> review, not from adding a new feature) and it fails for a previously
>>> "valid" revision, is the tag removed?
>>
>> There would probably be some sort of concept of "last known good". In
>> the above case, either the tag could be removed and/or a new valid
>> revision tagged. It was "good" up until that point, at least. :-)
>
> I would say that once a validation tag is created, it stays.  We  
> don't want to
> be in the business of going back through lots of tag history and  
> checking
> against new tests that didn't exist when the tags were originally  
> created.
>
After thinking about it more, I agree. There will always be bugs. We  
never claim that a given release is perfect, only that it doesn't have  
any issues currently known at the time it was tagged/released.

> Perhaps for a "last known good" tag we could re-check but I'm even  
> hesitent
> to do that.  It seems like a lot of extra work for little real gain.
>
You're probably correct.

-bw