[cfe-dev] [llvm-dev] test-suite: a new proposal for how to move forward to make "test-suite" more automatic, more flexible, and more maintainable, especially WRT reference outputs

[Renato Golin via cfe-dev]

On 6 October 2016 at 19:17, Sebastian Pop <sebpop.llvm at gmail.com> wrote:
> I'm also fine using any stable hashing function and link polybench
> tests against that.

The problem with hashes is that it only works when the results have to
be exact, which is not suitable for FP comparisons, thus would fail
under fp-contract=on.

To speed things up, I guess we could reduce the output via meaningful
aggregation, get a small set of FP numbers to compare against and set
an FP_TOLERANCE that makes sense around FP contraction and leave it
on.

Then we commit the change to fp-contract=on.

After that, we change the test harness to have two runs, one with
tolerance=0 and fp-contract=off, and the other with tolerance=delta
and fp-contract=on.

cheers,
--renato