[cfe-dev] [test-suite] making polybench/symm succeed with "-Ofast" and "-ffp-contract=on"

[Renato Golin via cfe-dev]

On 12 October 2016 at 14:26, Sebastian Pop <sebpop.llvm at gmail.com> wrote:
> Correct me if I misunderstood: you would be ok changing the
> reference output to exactly match the output of "-O0 -ffp-contract=off".

No, that's not at all what I said.

Matching identical outputs to FP tests makes no sense because there's
*always* an error bar.

The output of O0, O1, O2, O3, Ofast, Os, Oz should all be within the
boundaries of an average and its associated error bar.

By understanding what's the *expected* output and its associated error
range we can accurately predict what will be the correct
reference_output and the tolerance for each individual test.

Your solution 2 "works" because you're doing the matching yourself, in
the code, and for that, you pay the penalty of running it twice. But
it's not easy to control the tolerance, nor it's stable for all
platforms where we don't yet run the test suite.

My original proposal, and what I'm still proposing here, is to
understand the tests and make them right, by giving them proper
references and tolerances. If the output is too large, reduce/sample
in a way that doesn't increase the error ranges too much, enough to
keep the tolerance low, so we can still catch bugs in the FP
transformations.

cheers,
--renato