[llvm-dev] [test-suite] making the test-suite succeed with "-Ofast" and "-ffp-contract=on"

[Matthias Braun via llvm-dev]

> On Oct 7, 2016, at 6:46 PM, Hal Finkel via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> ----- Original Message -----
>> From: "Hal Finkel via llvm-dev" <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
>> To: "Sebastian Pop" <sebpop.llvm at gmail.com <mailto:sebpop.llvm at gmail.com>>
>> Cc: "Sebastian Paul Pop" <s.pop at samsung.com <mailto:s.pop at samsung.com>>, "llvm-dev" <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>, "Matthias Braun"
>> <matze at braunis.de <mailto:matze at braunis.de>>, "Clang Dev" <cfe-dev at lists.llvm.org <mailto:cfe-dev at lists.llvm.org>>, "nd" <nd at arm.com <mailto:nd at arm.com>>, "Abe Skolnik" <a.skolnik at samsung.com <mailto:a.skolnik at samsung.com>>
>> Sent: Friday, October 7, 2016 7:56:53 PM
>> Subject: Re: [llvm-dev] [test-suite] making the test-suite succeed with "-Ofast" and "-ffp-contract=on"
>> 
>> ----- Original Message -----
>>> From: "Sebastian Pop" <sebpop.llvm at gmail.com>
>>> To: "Renato Golin" <renato.golin at linaro.org>
>>> Cc: "Kristof Beyls" <Kristof.Beyls at arm.com>, "Sebastian Paul Pop"
>>> <s.pop at samsung.com>, "llvm-dev"
>>> <llvm-dev at lists.llvm.org>, "nd" <nd at arm.com>, "Abe Skolnik"
>>> <a.skolnik at samsung.com>, "Clang Dev"
>>> <cfe-dev at lists.llvm.org>, "Hal Finkel" <hfinkel at anl.gov>, "Stephen
>>> Canon" <scanon at apple.com>, "Matthias Braun"
>>> <matze at braunis.de>
>>> Sent: Friday, October 7, 2016 7:34:40 PM
>>> Subject: [test-suite] making the test-suite succeed with "-Ofast"
>>> and "-ffp-contract=on"
>>> 
>>> Hi,
>>> 
>>> I would like to provide a summary of the different proposals on how
>>> to
>>> fix the test-suite to make it succeed when specifying extra CFLAGS
>>> "-Ofast" and "-ffp-contract=on".  I would like to expose the issue
>>> and
>>> proposed ways to fix it to other potential reviewers that could
>>> provide extra feedback.  We also need to decide which proposal (or
>>> combination of) to implement and commit.
>>> 
>>> Proposal 1: https://reviews.llvm.org/D25277
>>> modify the CMakes to compile and run each of these benchmarks
>>> twice:
>>> once with added CFLAGS -ffp-contract=off.  Record on disk the full
>>> output of both runs and compare with FP_TOLERANCE.  Hash the output
>>> of
>>> the run with -ffp-contract=off and exact match against the
>>> reference
>>> output.
>>> 
>>> The good for Proposal 1:
>>> - changes contained in the build system: no change to the code of
>>> the
>>> benchmarks
>>> - runs benchmarks under an extra configuration with CFLAGS +=
>>> -ffp-contract=off
>>> 
>>> The bad for Proposal 1:
>>> - compilation time will double
>>> - running time on the device will double
>>> - build system is more complex
>>> - the build directory goes from 300M to 1.2G due to the extra
>>> reference outputs recorded under -ffp-contract=off,
>>> - when running test-suite over small devices it will cost 1G more
>>> transfer over the network.
>> 
>> I prefer proposal 1 (although, to be fair, it was something I
>> suggested). Being the the business of trying to heavily modify every
>> benchmark that does floating-point computation, as in proposal 2,
>> does not seem to scale well, and can't always be done regardless.
>> 
>> We can make some effort to reduce the size of the problems being
>> computed by some of the benchmarks (e.g. pollybench); I think that
>> is reasonable and will help with the extra space requirements. That
>> having been said, functionally speaking, our test suite is at least
>> an order of magnitude too small, and so my sympathy is somewhat
>> limited. We're going to have to find a way to execute the test suite
>> in stages on smaller devices to limit the peak usage, if not because
>> of this then because we've added a lot more test applications and
>> benchmarks in the future.
> 
> Another aspect to this is that we should have this kind of infrastructure for other purposes as well. We have a similar lack of testing for -ffast-math. We don't even do a good job of (i.e. have good buildbot coverage for) running the test suite @ -O1. -O2 and -O3 are much better tested. More regular testing at -O0 (especially with -g to pick up crashes in our debug-info generation logic) is needed as well.

I think we should rather just setup more build jobs that run different confiruations rather than modifying the testsuite to compile multiple configurations in a single run, as that keeps things simpler.
Chris and me also started collecting typical configurations in the test-suite/cmake/caches/ directory, I hope this will become a popular thing with more bot owners as it nicely documents what people are doing IMO and it makes it easy to describe the configuration used in bugreports / when reproducing issues on a personal machine.

- Matthias


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20161007/f05959c5/attachment.html>