[LLVMdev] Validating LLVM

[Bill Wendling]

On Nov 12, 2008, at 3:03 PM, David Greene wrote:

> On Wednesday 12 November 2008 00:33, Bill Wendling wrote:
>
>>
>> Here are my opinions:
>>
>> I like the idea of regular validation tagging. However, I think that
>> it should be as automated as possible. I'm worried that validation
>> testing will be pushed off of people's plates indefinitely – even for
>> people who care deeply about a particular platform.
>
> Yes, automation is key.  I think it is very possible to do with this  
> proposal.
>
>> Here are the minimal set of tests that should go into a validation
>> test. (All of these should be done in "Release" mode.)
>>
>> * Regression Tests - This catches obvious errors, but not major ones.
>
> For clarity, you mean "make check" on the LLVM tools, right?
>
Yeah. It's our first line of defense, so to speak. :-)

>> * Full Bootstrap of LLVM-GCC - LLVM-GCC is a complex program. It's  
>> our
>> second indication if something has gone horribly awry.
>
> Yep.
>
>> * Nightly Testsuite - A very good suite of tests; much more extensive
>> than a simple bootstrap.
>
> How does this differ from "make check" or llvm-test?
>
I may be wrong, but I think that not many people run the llvm-tests  
testsuite before their commits -- unless it's a major change requiring  
an "llc beta" check. Also, the nightly testsuite gives us measurements  
on performance, something that's certainly not done before most check- 
ins.

>> * LLVM-GCC Testsuite - Many thousands of great tests to test the many
>> facets of the compiler. WAY too few people run these.
>
> By this do you mean llvm-test or the testsuite that ships with gcc?   
> To
> my knowledge, LLVM has never passed the gcc testsuite ("make check"
> on llvm-gcc).
>
The testsuite that ships with gcc. Dale has been working on getting  
more and more of those tests passing. Some of the tests are invalid  
for LLVM (and should be identified and marked as such). However, it's  
a quite extensive testsuite, and we should do our best at limiting the  
number of failures in it.

>> As far as I know, Dale's the only one who's been slogging through the
>> LLVM-GCC testsuite, finding and fixing errors. I think that there are
>> still many failures that should be addressed.
>
> Depending on how you're defining LLVM-GCC I may also be running  
> those tests
> regularly.
>
Excellent! :-)

>> All four of the above should be run on at least a nightly basis (more
>> frequently for some, like the regression tests). Each of these are
>> automated, making that easy. If there are no regressions from the
>> above four, we could tag that revision as being potentially "valid".
>
> Right.  I would add one thing.  We want to run these suites with  
> Debug,
> Release, Release+Asserts and Debug+ExpensiveChecks builds.  No one
> but me seems to run Debug+ExpensiveChecks tests because I see things
> break regularly.  It's a valuable tool to find subtle C++ errors.
>
I agree with all of those. I also agree with Devang that we should run  
the tests with a "self hosted" LLVM, i.e., LLVM built with LLVM-GCC.

-bw