[LLVMdev] Validating LLVM

[David Greene]

On Monday 10 November 2008 22:17, John Regehr wrote:
> Lately our random C program generator has seemed quite successful at
> catching regressions in llvm-gcc that the test suite misses.  I'd suggest
> running some fixed number of random programs as part of the validation
> suite.  On a fastish quad core I can test about 25,000 programs in 24

The problem with random tests is that they're just that -- random.  You can't
have a known suite to validate with.  Now, if we gbenerate some tests
that cause things to fail and then add those to the LLVM test suite, I'd be
all for it.

> We've never tested llvm-gcc for x64 using random testing, doing this would
> likely turn up a nice crop of bugs.

Definitely.  Random testing is certainly useful.  Once random tests are added
to a testsuite, we can use them for validation.  But I wouldn't want to 
require a validation to pass some set of random tests that shifts each test
cycle.

                                                -Dave