[LLVMdev] Validating LLVM

[David Greene]

On Wednesday 12 November 2008 00:33, Bill Wendling wrote:

> > Please take a look and send feedback toi the list.  I'd like to get
> > the
> > process moving early in the 2.4 cycle.
>
> Hi Dave,
>
> Here are my opinions:
>
> I like the idea of regular validation tagging. However, I think that
> it should be as automated as possible. I'm worried that validation
> testing will be pushed off of people's plates indefinitely – even for
> people who care deeply about a particular platform.

Yes, automation is key.  I think it is very possible to do with this proposal.

> Here are the minimal set of tests that should go into a validation
> test. (All of these should be done in "Release" mode.)
>
> * Regression Tests - This catches obvious errors, but not major ones.

For clarity, you mean "make check" on the LLVM tools, right?

> * Full Bootstrap of LLVM-GCC - LLVM-GCC is a complex program. It's our
> second indication if something has gone horribly awry.

Yep.

> * Nightly Testsuite - A very good suite of tests; much more extensive
> than a simple bootstrap.

How does this differ from "make check" or llvm-test?

> * LLVM-GCC Testsuite - Many thousands of great tests to test the many
> facets of the compiler. WAY too few people run these.

By this do you mean llvm-test or the testsuite that ships with gcc?  To
my knowledge, LLVM has never passed the gcc testsuite ("make check"
on llvm-gcc).

> As far as I know, Dale's the only one who's been slogging through the
> LLVM-GCC testsuite, finding and fixing errors. I think that there are
> still many failures that should be addressed.

Depending on how you're defining LLVM-GCC I may also be running those tests 
regularly.

> All four of the above should be run on at least a nightly basis (more
> frequently for some, like the regression tests). Each of these are
> automated, making that easy. If there are no regressions from the
> above four, we could tag that revision as being potentially "valid".

Right.  I would add one thing.  We want to run these suites with Debug, 
Release, Release+Asserts and Debug+ExpensiveChecks builds.  No one
but me seems to run Debug+ExpensiveChecks tests because I see things
break regularly.  It's a valuable tool to find subtle C++ errors.

                                                 -Dave