[PATCH] D24925: [testsuite] turn fp-contract off for ARM because output checking is not flexible enough

[Sanjay Patel via llvm-commits]

spatel added a comment.

In https://reviews.llvm.org/D24925#552706, @MatzeB wrote:

> 2. Is there a list of broken benchmarks somewhere? I don't see anyone doing numerical analysis on the test-suite benchmarks, what you can today though is telling the test-suite to use fpcmp for comparing the results and set some absolute or relative tolerances.


A first pass at broken benchmarks is anything that failed on the AArch bots with the fp-contract patch:

FAIL: MultiSource/Applications/oggenc/oggenc.execution_time (490 of 2445)
FAIL: MultiSource/Benchmarks/MiBench/telecomm-FFT/telecomm-fft.execution_time (491 of 2445)
FAIL: MultiSource/Benchmarks/VersaBench/beamformer/beamformer.execution_time (492 of 2445)
FAIL: SingleSource/Benchmarks/Linpack/linpack-pc.execution_time (493 of 2445)
FAIL: SingleSource/Benchmarks/Misc-C++/Large/sphereflake.execution_time (494 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/datamining/correlation/correlation.execution_time (495 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/datamining/covariance/covariance.execution_time (496 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/2mm/2mm.execution_time (497 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/3mm/3mm.execution_time (498 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/atax/atax.execution_time (499 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/bicg/bicg.execution_time (500 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/cholesky/cholesky.execution_time (501 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/gemver/gemver.execution_time (502 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/gesummv/gesummv.execution_time (503 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/mvt/mvt.execution_time (504 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/symm/symm.execution_time (505 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/trisolv/trisolv.execution_time (506 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/kernels/trmm/trmm.execution_time (507 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/linear-algebra/solvers/gramschmidt/gramschmidt.execution_time (508 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/stencils/adi/adi.execution_time (509 of 2445)
FAIL: SingleSource/Benchmarks/Polybench/stencils/fdtd-2d/fdtd-2d.execution_time (510 of 2445)


https://reviews.llvm.org/D24925