[LLVMdev] Validating LLVM

[John Regehr]

> to a testsuite, we can use them for validation.  But I wouldn't want to
> require a validation to pass some set of random tests that shifts each test
> cycle.

This is easy to fix: just specify a starting seed for the PRNG.

However I think you should get past your prejudice against tests that 
shift each cycle, since changing tests have the advantage of increased 
test coverage.  Different parts of a test suite have different purposes, 
and of course random programs would not replace any part of the existing 
collection of fixed test cases.  I woudn't be making this argument if I 
hadn't seen for myself how one week random testing gives you nothing, the 
next week a whole pile of previously unknown failures.

Alternatively we are working to generalize our program generator a bit so 
that it does a DFS or BFS to generate all programs smaller than some size 
bound (obviously we need to fudge on integer constants, for example by 
picking from a predetermined set of interesting constants).  Once we do 
this it may be worth adding the resulting test programs to LLVM's test 
suite.

John Regehr