[LLVMdev] Validation Criteria

[David Greene]

As we begin designing the proposed validation process, we should think about 
what the criteria for a successful validation should be.

In the initial proposal I suggested the "make check" on llvm should pass as 
should the tests in llvm-test.

According to Tanya, not all of llvm-test passes.  Do we have a sense of how 
far away from a full passing llvm-test we are?

I also know that a lot of work has gone into making the llvm-gcc DejaGNU tests 
pass.  Where are we in that process?

Initially, I'll throw out the idea that if llvm passes its internal tests 
("make check") and successfully bootstraps llvm-gcc-4.2 we can consider the 
validation successful.  Does that sound like a reasonable starting point?  The 
assumption here is that as regressions in other tests (llvm-test, SPEC, etc.) 
are found, testcases will be submitted to the llvm testsuite.

Once llvm-test passes, we can add it to the list of required tests to pass.  
Ditto llvm-gcc-4.2's tests and anything else we might want to add.

Note that as tests get added to existing testsuites, we'll require the new 
tests to pass as well.  So once llvm-test is added to the validation 
criteria, anything added to llvm-test after that will also have to pass in 
order for a validation to succeed.  I don't want to get into the business of 
tracking individual test regressions, XFAIL'ing things that run outside 
DejaGNU, etc.  It's a lot of extra work for not much gain, IMHO.

Would anyone want to add other criteria?

                                            -Dave