[LLVMdev] Validating LLVM

[Andrew Beyer]

>> to a testsuite, we can use them for validation.  But I wouldn't want to
>> require a validation to pass some set of random tests that shifts each test
>> cycle.
>
> This is easy to fix: just specify a starting seed for the PRNG.

...which defeats much of the point of random testing.

> However I think you should get past your prejudice against tests that
> shift each cycle, since changing tests have the advantage of increased
> test coverage.  Different parts of a test suite have different purposes,
> and of course random programs would not replace any part of the existing
> collection of fixed test cases.  I woudn't be making this argument if I
> hadn't seen for myself how one week random testing gives you nothing, the
> next week a whole pile of previously unknown failures.

I don't think anyone is arguing against the utility of random test
generation, the issue is that the results aren't really appropriate
for validation where you are trying to make a comparison between
builds. A system I've used previously was to run and report randomly
generated tests along with validation testing, but not consider the
random test results when tagging a build as valid. Instead, when
random tests failed, the generated test case was saved, and could be
added (preferably in a reduced form) as a static test for future
validation runs. That way you get the benefits of random testing
without spurious changes in validation status dependent on randomly
generated tests.