<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Oct 7, 2016, at 6:46 PM, Hal Finkel via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">----- Original Message -----</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">From: "Hal Finkel via llvm-dev" <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>><br class="">To: "Sebastian Pop" <<a href="mailto:sebpop.llvm@gmail.com" class="">sebpop.llvm@gmail.com</a>><br class="">Cc: "Sebastian Paul Pop" <<a href="mailto:s.pop@samsung.com" class="">s.pop@samsung.com</a>>, "llvm-dev" <<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>>, "Matthias Braun"<br class=""><<a href="mailto:matze@braunis.de" class="">matze@braunis.de</a>>, "Clang Dev" <<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a>>, "nd" <<a href="mailto:nd@arm.com" class="">nd@arm.com</a>>, "Abe Skolnik" <<a href="mailto:a.skolnik@samsung.com" class="">a.skolnik@samsung.com</a>><br class="">Sent: Friday, October 7, 2016 7:56:53 PM<br class="">Subject: Re: [llvm-dev] [test-suite] making the test-suite succeed with "-Ofast" and "-ffp-contract=on"<br class=""><br class="">----- Original Message -----<br class=""><blockquote type="cite" class="">From: "Sebastian Pop" <<a href="mailto:sebpop.llvm@gmail.com" class="">sebpop.llvm@gmail.com</a>><br class="">To: "Renato Golin" <<a href="mailto:renato.golin@linaro.org" class="">renato.golin@linaro.org</a>><br class="">Cc: "Kristof Beyls" <<a href="mailto:Kristof.Beyls@arm.com" class="">Kristof.Beyls@arm.com</a>>, "Sebastian Paul Pop"<br class=""><<a href="mailto:s.pop@samsung.com" class="">s.pop@samsung.com</a>>, "llvm-dev"<br class=""><<a href="mailto:llvm-dev@lists.llvm.org" class="">llvm-dev@lists.llvm.org</a>>, "nd" <<a href="mailto:nd@arm.com" class="">nd@arm.com</a>>, "Abe Skolnik"<br class=""><<a href="mailto:a.skolnik@samsung.com" class="">a.skolnik@samsung.com</a>>, "Clang Dev"<br class=""><<a href="mailto:cfe-dev@lists.llvm.org" class="">cfe-dev@lists.llvm.org</a>>, "Hal Finkel" <<a href="mailto:hfinkel@anl.gov" class="">hfinkel@anl.gov</a>>, "Stephen<br class="">Canon" <<a href="mailto:scanon@apple.com" class="">scanon@apple.com</a>>, "Matthias Braun"<br class=""><<a href="mailto:matze@braunis.de" class="">matze@braunis.de</a>><br class="">Sent: Friday, October 7, 2016 7:34:40 PM<br class="">Subject: [test-suite] making the test-suite succeed with "-Ofast"<br class="">and "-ffp-contract=on"<br class=""><br class="">Hi,<br class=""><br class="">I would like to provide a summary of the different proposals on how<br class="">to<br class="">fix the test-suite to make it succeed when specifying extra CFLAGS<br class="">"-Ofast" and "-ffp-contract=on". I would like to expose the issue<br class="">and<br class="">proposed ways to fix it to other potential reviewers that could<br class="">provide extra feedback. We also need to decide which proposal (or<br class="">combination of) to implement and commit.<br class=""><br class="">Proposal 1: <a href="https://reviews.llvm.org/D25277" class="">https://reviews.llvm.org/D25277</a><br class="">modify the CMakes to compile and run each of these benchmarks<br class="">twice:<br class="">once with added CFLAGS -ffp-contract=off. Record on disk the full<br class="">output of both runs and compare with FP_TOLERANCE. Hash the output<br class="">of<br class="">the run with -ffp-contract=off and exact match against the<br class="">reference<br class="">output.<br class=""><br class="">The good for Proposal 1:<br class="">- changes contained in the build system: no change to the code of<br class="">the<br class="">benchmarks<br class="">- runs benchmarks under an extra configuration with CFLAGS +=<br class="">-ffp-contract=off<br class=""><br class="">The bad for Proposal 1:<br class="">- compilation time will double<br class="">- running time on the device will double<br class="">- build system is more complex<br class="">- the build directory goes from 300M to 1.2G due to the extra<br class="">reference outputs recorded under -ffp-contract=off,<br class="">- when running test-suite over small devices it will cost 1G more<br class="">transfer over the network.<br class=""></blockquote><br class="">I prefer proposal 1 (although, to be fair, it was something I<br class="">suggested). Being the the business of trying to heavily modify every<br class="">benchmark that does floating-point computation, as in proposal 2,<br class="">does not seem to scale well, and can't always be done regardless.<br class=""><br class="">We can make some effort to reduce the size of the problems being<br class="">computed by some of the benchmarks (e.g. pollybench); I think that<br class="">is reasonable and will help with the extra space requirements. That<br class="">having been said, functionally speaking, our test suite is at least<br class="">an order of magnitude too small, and so my sympathy is somewhat<br class="">limited. We're going to have to find a way to execute the test suite<br class="">in stages on smaller devices to limit the peak usage, if not because<br class="">of this then because we've added a lot more test applications and<br class="">benchmarks in the future.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Another aspect to this is that we should have this kind of infrastructure for other purposes as well. We have a similar lack of testing for -ffast-math. We don't even do a good job of (i.e. have good buildbot coverage for) running the test suite @ -O1. -O2 and -O3 are much better tested. More regular testing at -O0 (especially with -g to pick up crashes in our debug-info generation logic) is needed as well.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div></div>I think we should rather just setup more build jobs that run different confiruations rather than modifying the testsuite to compile multiple configurations in a single run, as that keeps things simpler.<div class="">Chris and me also started collecting typical configurations in the test-suite/cmake/caches/ directory, I hope this will become a popular thing with more bot owners as it nicely documents what people are doing IMO and it makes it easy to describe the configuration used in bugreports / when reproducing issues on a personal machine.</div><div class=""><br class=""></div><div class="">- Matthias</div><div class=""><div class=""><br class=""></div><div class=""><br class=""></div></div></body></html>