<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: arial,helvetica,sans-serif; font-size: 10pt; color: #000000'><br><hr id="zwchr"><blockquote id="DWT3977" style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>"Matthias Braun" <mbraun@apple.com><br><b>To: </b>"Hal Finkel" <hfinkel@anl.gov><br><b>Cc: </b>"llvm-dev" <llvm-dev@lists.llvm.org>, "cfe-dev" <cfe-dev@lists.llvm.org>, "Abe Skolnik" <a.skolnik@samsung.com><br><b>Sent: </b>Thursday, September 29, 2016 6:42:55 PM<br><b>Subject: </b>Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs and more-optimized builds that may have different outputs due to FP optimizations<br><br>
<br class=""><div><blockquote class=""><div class="">On Sep 29, 2016, at 4:35 PM, Hal Finkel <<a href="mailto:hfinkel@anl.gov" class="" target="_blank">hfinkel@anl.gov</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline ! important;" class=""><hr id="zwchr"></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><blockquote style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class="">From: "Matthias Braun via cfe-dev" <<a href="mailto:cfe-dev@lists.llvm.org" class="" target="_blank">cfe-dev@lists.llvm.org</a>><br class="">To: "Abe Skolnik" <<a href="mailto:a.skolnik@samsung.com" class="" target="_blank">a.skolnik@samsung.com</a>><br class="">Cc: "llvm-dev" <<a href="mailto:llvm-dev@lists.llvm.org" class="" target="_blank">llvm-dev@lists.llvm.org</a>>, "cfe-dev" <<a href="mailto:cfe-dev@lists.llvm.org" class="" target="_blank">cfe-dev@lists.llvm.org</a>><br class="">Sent: Thursday, September 29, 2016 6:20:09 PM<br class="">Subject: Re: [cfe-dev] [llvm-dev] improving test-suite`s FP subtests to be able to compare both exact-match outputs<br class="">and more-optimized builds that may have different outputs due to FP optimizations<br class=""><br class=""><br class=""><blockquote class="">On Sep 29, 2016, at 3:59 PM, Abe Skolnik <<a href="mailto:a.skolnik@samsung.com" class="" target="_blank">a.skolnik@samsung.com</a>><br class="">wrote:<br class=""><br class="">Dear all,<br class=""><br class="">I would like some help, please, with implementing Hal`s excellent<br class="">suggestion, which I have reworded as below. Hal has confirmed a<br class="">previous version of my rewording as a correct interpretation. [I<br class="">made minor changes since then, e.g. for grammar.]<br class=""><br class="">[Abe wrote:]<br class=""><br class=""><blockquote class=""><blockquote class="">I think you [Hal] are suggesting something like this:<br class=""></blockquote></blockquote><br class=""><blockquote class=""><blockquote class=""> 1) compile the program with FP fusion off,<br class=""> run the program, capture the output and save it,<br class=""> hash it and compare it against the reference hash.<br class=""></blockquote></blockquote><br class=""><blockquote class=""><blockquote class=""> 2) if comparison against the reference hash says "not equal",<br class=""> fail the test and stop [i.e. stop testing this particular<br class=""> subtest]<br class=""></blockquote></blockquote><br class=""><blockquote class=""><blockquote class=""> 3) compile the program with FP fusion on/"fast", capture the<br class=""> output,<br class=""> compare it using "fpcmp" and some positive tolerance against<br class=""> the<br class=""> output of the non-fusion build of the same source code;<br class=""> fail only if outside the tolerance limit[s]<br class=""></blockquote></blockquote></blockquote><br class="">Currently the test-suite works by first building the executable and<br class="">then running them on a set of inputs. Having a multi-step thingy<br class="">does not fit that.<br class=""><br class="">Having said that you could possibly just build two variants first and<br class="">use different comparison steps for each. That at least fits the<br class="">model but is still a precedent and requires some deeper test-suite<br class="">buildsystem hacking.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline ! important;" class="">I think makes sense. We can have the fp-contract-off target and the fp-contract-default target, and use set_target_properties to add -ffp-contract=off to the compile_flags for the former. Then we just need to add the output from the default target as a generated file, and add a dependency on it to the fp-contract-default comparison step. Does that sound right?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""></div></blockquote><div>in a cmake build that would be the right thing to do. The test-suite typically uses a bunch of wrappers above all that so the cmakefiles look more like the old makefiles did... You can probably just duplicate everything in the current beam/CMakeLists.txt and just set a different name for the "PROG" variable and adjust the CFLAGS as wanted. However that needs some careful testing to make sure those two targets are indeed completely independent and don't override each others file.</div><br class=""><blockquote class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; float: none; display: inline ! important;" class="">Are the buildbots still using the makefile-based system, or are they all on the cmake-based system now?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""></div></blockquote><div><br class=""></div></div>The scripts in the zorg repository appear to be using "lnt runtest nt" (= makefile system) rather than "lnt runtest test-suite" (=cmake/lit test-suite). So the majority of bots is still on the makefiles I assume :-(</blockquote>Is there any reason we should not start transitioning that now?<br><br> -Hal<br><blockquote style="border-left: 2px solid rgb(16, 16, 255); margin-left: 5px; padding-left: 5px; color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><div class=""><br class=""></div><div class="">- Matthias</div></blockquote><br><br><br>-- <br><div><span name="x"></span>Hal Finkel<br>Lead, Compiler Technology and Programming Languages<br>Leadership Computing Facility<br>Argonne National Laboratory<span name="x"></span><br></div></div></body></html>