<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">> We've hit at least one failure when porting because the hashing means that miniscule differences in standard library implementations are not tolerated.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">How did you address that? I would expect there to be many small differences between standard library implementations and across different platforms.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Wherever the language standard doesn’t require correctly rounded results there is a potential for small variations. I think this problem becomes more prominent with tests that execute on GPU and similar hardware.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Hubert Tong <hubert.reinterpretcast@gmail.com> <br>
<b>Sent:</b> Wednesday, June 23, 2021 5:37 PM<br>
<b>To:</b> Kaylor, Andrew <andrew.kaylor@intel.com>; amykibm@gmail.com<br>
<b>Cc:</b> llvm-dev@lists.llvm.org<br>
<b>Subject:</b> Re: [llvm-dev] Floating point variance in the test suite<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Jun 23, 2021 at 1:57 PM Kaylor, Andrew via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi everyone,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’d like to restart an old discussion about how to handle floating point variance in the LLVM test suite. This was brought up years ago by Sebastian Pop (<a href="https://lists.llvm.org/pipermail/llvm-dev/2016-October/105730.html" target="_blank">https://lists.llvm.org/pipermail/llvm-dev/2016-October/105730.html</a>)
and Sebastian did some work to fix things at that time, but I’ve discovered recently that a lot of things aren’t working the way they are supposed to, and some fundamental problems were never addressed.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The basic issue is that at least some tests fail if the floating point calculations don’t exactly match the reference results. In 2016, Sebastian attempted to modify the Polybench
tests to allow some tolerance, but that attempt was not entirely successful. Other tests don’t seem to have a way to handle this. I don’t know how many.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Melanie Blower has been trying for some time now to commit a change that would make the fp-contract=on the default setting for clang (as the documentation says it is), but this
change caused failures in the test suite. Melanie recently committed a change (<a href="https://reviews.llvm.org/rT24550c3385e8e3703ed364e1ce20b06de97bbeee" target="_blank">https://reviews.llvm.org/rT24550c3385e8e3703ed364e1ce20b06de97bbeee</a>) which overrides
the default and sets fp-contract=off for the failing tests, but this is not a good long term solution, as it leaves fp contraction untested (though apparently it has been for quite some time).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The test suite attempts to handle floating point tests in several different ways (depending on the test configuration):<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">1. Tests write floating point results to a text file and the fpcmp utility is used to compare them against reference output. The method allows absolute and relative tolerance to
be specified, I think.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2. Tests write floating point results to a text file which is then hashed and compared against a reference hash value.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">3. (Sebastian’s 2016 change) Tests are run with two kernels, one which is compiled with FMA explicitly disabled and one with the test suite configured options. The results of these
kernels are compared with a specified tolerance, then the FMA-disabled results are written to a text file, hashed and compared to a reference output.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’ve discovered a few problems with this.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">First, many of the tests are producing hashed results but using fpcmp to compare the hash values. I created
<a href="https://bugs.llvm.org/show_bug.cgi?id=50818" target="_blank">https://bugs.llvm.org/show_bug.cgi?id=50818</a> to track this problem. I don’t know the configuration well enough to fix it, but it seems like it should be a simple problem. If the hash values
match, it works (for the wrong reasons). It doesn’t allow any FP tolerance, but that seems to be expected (hashing and FP tolerance cannot be combined in the configuration files). Personally, I don’t like the use of hashing with floating point results, so
I’d like to get rid of method 2.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We've hit at least one failure when porting because the hashing means that miniscule differences in standard library implementations are not tolerated.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Second, when the third method is used FMA is being disabled using the STDC FP_CONTRACT pragma. Currently, LLVM does not respect this pragma when fp-contract=fast is used. This seems
to be accepted behavior, but in my opinion it’s obviously wrong. A new setting was added recently for fp-contract=fast-honor-pragmas. I would very much like to see this work the other way -- by default fp-contract=fast should honor the pragmas, and if someone
needs a setting that doesn’t that can be added. In any event, the relevant information here is that Sebastian’s FMA disabling solution doesn’t work for fp-contract=fast. Both kernels are compiled with FMA enabled, so their results match, but the test fails
the hash comparison because the “FMA disabled” kernel really had FMA enabled.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Third, when the third method is used, it’s checking the intermediate results using “FP_ABSTOLERANCE” and in some cases FMA exceeds the tolerance currently configured. For example,
in the Polybench symm test I got this output:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">A[5][911] = 85644607039.746628 and B[5][911] = 85644607039.746643 differ more than FP_ABSTOLERANCE = 0.000010<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The difference there looks pretty reasonable to me, but because we’re looking for a minimal absolute difference, the test failed. Incidentally, the test failed with a message that
said “fpcmp-target: FP Comparison failed, not a numeric difference between '0' and 'b'” because the above output got hashed and compared to a reference hash value using the tool that expected both hash values to be floating point values. The LNT bots don’t
even give you this much information though, as far as I can tell, they just tell you the test failed. But I digress.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Finally, a few of the Polybench tests are intending to use the third method above but aren’t actually calling the “strict” kernel so they fail with fp-contract=on.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">So, that’s what I know. There is an additional problem that has never been addressed regarding running the test suite with fast-math enabled.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Now I suppose I should say what kind of feedback I’m looking for.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I guess the first thing I want is to find out who is interested in the results and behavior of these tests. I assume there are people who care, but they get touched so infrequently
and are in such a bad state that I don’t know if anyone is even paying attention beyond trying to keep the bots green. I don’t want to spend a lot of time cleaning up tests that aren’t useful to anyone just because they happen to be in the test suite.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Beyond that, I’d like to get general feedback on the strategy that we should be adopting in the test suite. In the earlier discussion of this issue, the consensus seemed to be that
the problems should be addressed on a case-by-case basis, doing the “right thing” for each test. This kind of implies a level of test ownership. I would like input for each test from someone who cares about that test.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Finally, I’d like to hear some general opinions on whether the tests we have now are useful and sufficient for floating point behavior.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Andy<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" target="_blank">https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><o:p></o:p></p>
</div>
</body>
</html>