[LLVMdev] Validating LLVM

[John Regehr]

Lately our random C program generator has seemed quite successful at 
catching regressions in llvm-gcc that the test suite misses.  I'd suggest 
running some fixed number of random programs as part of the validation 
suite.  On a fastish quad core I can test about 25,000 programs in 24 
hours.  Our hacked valgrind (which looks for volatile miscompilations) is 
a bottleneck, leaving this out would speed up the process considerably.

We've never tested llvm-gcc for x64 using random testing, doing this would 
likely turn up a nice crop of bugs.

I just started a random test run of llvm-gcc 2.0-2.4 that should provide 
some interesting quantitative results comparing these compilers in terms 
of crashes, volatile miscompilations, and regular miscompilations. 
However it may take a month or so to get statistical significance since 
2.3 and 2.4 have quite low failure rates.

John Regehr