[LLVMdev] Validating LLVM

Andrew Beyer beyer.andrew at gmail.com
Tue Nov 11 12:29:10 PST 2008

>> to a testsuite, we can use them for validation.  But I wouldn't want to
>> require a validation to pass some set of random tests that shifts each test
>> cycle.
> This is easy to fix: just specify a starting seed for the PRNG.

...which defeats much of the point of random testing.

> However I think you should get past your prejudice against tests that
> shift each cycle, since changing tests have the advantage of increased
> test coverage.  Different parts of a test suite have different purposes,
> and of course random programs would not replace any part of the existing
> collection of fixed test cases.  I woudn't be making this argument if I
> hadn't seen for myself how one week random testing gives you nothing, the
> next week a whole pile of previously unknown failures.

I don't think anyone is arguing against the utility of random test
generation, the issue is that the results aren't really appropriate
for validation where you are trying to make a comparison between
builds. A system I've used previously was to run and report randomly
generated tests along with validation testing, but not consider the
random test results when tagging a build as valid. Instead, when
random tests failed, the generated test case was saved, and could be
added (preferably in a reduced form) as a static test for future
validation runs. That way you get the benefits of random testing
without spurious changes in validation status dependent on randomly
generated tests.

More information about the llvm-dev mailing list