[llvm-dev] [test-suite] making polybench/symm succeed with "-Ofast" and "-ffp-contract=on"

[Sebastian Pop via llvm-dev]

On Wed, Oct 12, 2016 at 9:35 AM, Renato Golin <renato.golin at linaro.org> wrote:
> On 12 October 2016 at 14:26, Sebastian Pop <sebpop.llvm at gmail.com> wrote:
>> Correct me if I misunderstood: you would be ok changing the
>> reference output to exactly match the output of "-O0 -ffp-contract=off".
>
> No, that's not at all what I said.
>
> Matching identical outputs to FP tests makes no sense because there's
> *always* an error bar.
>
> The output of O0, O1, O2, O3, Ofast, Os, Oz should all be within the
> boundaries of an average and its associated error bar.
>
> By understanding what's the *expected* output and its associated error
> range we can accurately predict what will be the correct
> reference_output and the tolerance for each individual test.
>
> Your solution 2 "works" because you're doing the matching yourself, in
> the code, and for that, you pay the penalty of running it twice. But
> it's not easy to control the tolerance, nor it's stable for all
> platforms where we don't yet run the test suite.
>
> My original proposal, and what I'm still proposing here, is to
> understand the tests and make them right, by giving them proper
> references and tolerances.

There is also the problem that I documented for 5 of the benchmarks
where the error margin between -Ofast and "-O0 -ffp-contract=off" is too big:

polybench/linear-algebra/kernels/symm, FP_ABSTOLERANCE=1e1
polybench/linear-algebra/solvers/gramschmidt, FP_ABSTOLERANCE=1e0
polybench/medley/reg_detect, FP_ABSTOLERANCE=1e4
polybench/stencils/adi, FP_ABSTOLERANCE=1e4

These differences come from the fact that these benchmarks
contain reductions of 1000+ values. The reductions are cumulating
errors, making end result diverge as the problem size increases.

> If the output is too large, reduce/sample
> in a way that doesn't increase the error ranges too much, enough to
> keep the tolerance low, so we can still catch bugs in the FP
> transformations.
>
> cheers,
> --renato