[llvm-dev] [test-suite] making polybench/symm succeed with "-Ofast" and "-ffp-contract=on"

[Renato Golin via llvm-dev]

On 12 October 2016 at 13:04, Sebastian Pop <sebpop.llvm at gmail.com> wrote:
> The other problem is the reference output does not match
> at "-O0 -ffp-contract=off". It might be that the reference output was recorded
> at "-O3 -ffp-contract=off". I think that this hides either a compiler
> bug or a test bug.

Ah, yes! You mentioned before and I forgot to reply, you're absolutely right.

If the tolerance is zero, then it's "ok" to "fail" at O0, because
whatever O3 produces is "some" version of the expected value +- some
delta. The error is expecting the tolerance to be zero (or smaller
than delta).

My point, since the beginning, has been to understand what the
expected value (with its inherent error bars), and make that the
reference output. Only then the test will be meaningful *and*
accurate.

But there are so many overloaded terms in this conversation that it's
really hard to get a point across without going to great lengths to
explain each one. :)

cheers,
--renato

PS: the term "accurate" above is meant to "accurately test the
expected error ranges the compiler is allowed to produce", not that
the test will have a lower error bar. It demonstrates the term
overloading quite well. :)