[PATCH] D25277: [test-suite] Run FP tests twice with -ffp-contract=off/on

[Matthias Braun via llvm-commits]

> On Oct 5, 2016, at 2:14 PM, Abe Skolnik via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> 
> [Matthias Braun wrote:]
> 
>> What I am hoping for/asking about is a way to perform some sort of arithmetic inside the benchmark so that we only need to print very few numbers.
> 
> Yes, but that requires domain-expert knowledge, which, as I wrote in an earlier email, I want to keep my hands out of.  Other people are welcome to contribute, and I`ll be happy to _help_ them if my help will be needed, but I`m not volunteering to make my entire work-life all about reading the source code for each test in LLVM`s test-suite that has a large amount of output. I think I might get fired if I did that without mgmt. approval, and I think the probability of my boss approving me working on that and only that for the next 6 months or so is -- to quote my favorite author of fiction -- "as near to nothing as makes no odds".
Sure, this was not targeted at you or asking you to do that. Though admittedly your employer (or the llvm community?) asked for -ffp-contract=on being the default and with that decision we introduced nondeterminism which I am sure some people will have to pay for by hunting down supposed bugs that turn out to be floatingpoint inaccuracies; I expect the test-suite to only be the tip of the iceberg here... Anyway that discussion seems to be over, so we should rather find ways to deal with it...

I was mainly wondering here whether there may be a sensible generic mechanism to combine a list of floatingpoint numbers; The answer to that may very well be no.
As far as I am concerned pushing the -ffp-contract=off switches to the benchmarks would have been enough and lesser evil compared to complicating the test-suite infrastructure further which has it's own problems.

> 
> FYI for all reading this: I humbly request that contributions along the lines of the above wait until Sebastian and I have sufficiently improved the test-suite that it is well-stabilized again.  We have some large changes in the works, and if accepted then there will be large improvements in the way the non-legacy harness works.
As I spent quite a bit of work on the current cmake/lit testsuite infrastructure I would be interested to hear some more details of what you are working on ;-) I don't plan big changes in the near future except for an additionol plugin to collect the llvm statistics generated by the new -save-stats clang flag...

- Matthias