[LLVMdev] Validating LLVM

[Bill Wendling]

On Nov 10, 2008, at 12:59 PM, David Greene wrote:

> Back during the LLVM developer's meeting, I talked with some of you  
> about a
> proposal to "validate" llvm.  Now that 2.4 is almost out the door,  
> it seems a
> good time to start that discussion.
>
> I've written up a detailed proposal and attached it to this  
> message.  The goal
> is to ease LLVM use by third parties.  We've got consideral  
> experience with
> LLVM and the community development model here and see its benefits  
> as well as
> challenges.  This proposal attempts to address what I feel is the main
> challenge: testing and stability of LLVM between releases.
>
> Please take a look and send feedback toi the list.  I'd like to get  
> the
> process moving early in the 2.4 cycle.
>
Hi Dave,

Here are my opinions:

I like the idea of regular validation tagging. However, I think that  
it should be as automated as possible. I'm worried that validation  
testing will be pushed off of people's plates indefinitely – even for  
people who care deeply about a particular platform.

Here are the minimal set of tests that should go into a validation  
test. (All of these should be done in "Release" mode.)

* Regression Tests - This catches obvious errors, but not major ones.
* Full Bootstrap of LLVM-GCC - LLVM-GCC is a complex program. It's our  
second indication if something has gone horribly awry.
* Nightly Testsuite - A very good suite of tests; much more extensive  
than a simple bootstrap.
* LLVM-GCC Testsuite - Many thousands of great tests to test the many  
facets of the compiler. WAY too few people run these.

As far as I know, Dale's the only one who's been slogging through the  
LLVM-GCC testsuite, finding and fixing errors. I think that there are  
still many failures that should be addressed.

All four of the above should be run on at least a nightly basis (more  
frequently for some, like the regression tests). Each of these are  
automated, making that easy. If there are no regressions from the  
above four, we could tag that revision as being potentially "valid".

-bw