<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">While reviewing a recent clang documentation change, I became aware of an issue with the way that clang is handling FP denormals. There is currently some support for variations in the way denormals are handled, but it isn't consistent across
architectures and generally feels kind of half-baked. I'd like to discuss possible solutions to this problem.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">First, there is a clang command line option: <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> -fdenormal-fp-math=<arg><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Select which denormal numbers the code is permitted to require.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Valid values are: ieee, preserve-sign, and positive-zero, which<o:p></o:p></p>
<p class="MsoNormal"> correspond to IEEE 754 denormal numbers, the sign of a flushed-to-zero<o:p></o:p></p>
<p class="MsoNormal"> number is preserved in the sign of 0, denormals are flushed to positive<o:p></o:p></p>
<p class="MsoNormal"> zero, respectively.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">A quick survey of the code leads me to believe this has no effect for targets other than ARM. For X86 targets we may want different options. I'll say more about that below. The wording of the documentation is sufficiently ambiguous that
I’m not entirely certain whether it is intended to control the target hardware or just the optimizer.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In addition, when either -Ofast or -ffast-math is used, we attempt to link 'crtfastmath.o' if it can be found. For X86 targets, this object file adds a static constructor that sets the DAZ and FTZ bits of the MXCSR register. I expect that
it has analogous behavior for other architectures when it is available. This object file is typically available on Linux systems, possibly also with things like MinGW. If it isn't found, the denomral control flags will be left in their default state.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There is also a CUDA-specific option, -f[no-]cuda-flush-denormals-to-zero. I don't know how this is implemented, but the documentation says it is specific to CUDA device mode.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Finally, there is an OpenCL-specific option, -cl-denorms-are-zero. Again, I don't know how it is implemented.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So.... I'd like to talk about how we can corral all of this into some interface that is consistent (or at least consistently sensible) across architectures.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The problems I see are:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">1. -fdenormal-fp-math needs to handle all scenarios needed by all architectures (or needs to be limited to a common subset).<o:p></o:p></p>
<p class="MsoNormal">2. -fdenormal-fp-math needs to be reconciled with -ffast-math and its variants.<o:p></o:p></p>
<p class="MsoNormal">3. -fdenormal-fp-math needs to be consistent about whether or not it imposes hardware changes when applicable.<o:p></o:p></p>
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoNormal">I can only really speak to X86, so I'll say a few words about that to start the discussion.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The current choices for -fdenormal-fp-math are: ieee, preserve-sign, and positive-zero. With X86, you get ieee behavior if neither DAZ or FTZ are set. If FTZ is set you get 'preserve sign' behavior -- i.e. denormal results are flushed to
zero and the sign of the result is kept. There is no way to get 'positive zero' behavior with X86. At the hardware level, modern X86 processors have separate controls for ftz (results are flushed to zero) and daz (inputs are flushed to zero before calculations),
but I doubt that they are used independently often enough to distinguish them at the command line option level.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Also, any X87 instructions that happen to be generated (such as if the code contains 'long double' data on Linux) will ignore the ftz and daz settings. There are some early Pentium 4 processors that don't support 'daz' but I hope we can
safely ignore that fact.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Linking in crtfastmath.o when -Ofast or -ffast-math are used is consistent with GCC's behavior. However, it implicitly ignores -fdenormal-fp-math, which GCC doesn't have. In most cases if a user sets a fast math option they probably also
want DAZ and FTZ, but there might be some reason why an advanced user would want to treat them separately. This can be done with intrinsics, of course, but if we have an option to control it, we should respect that option. Also, it is possible to construct
fast math behavior cafeteria-style (i.e. setting some fast math flags and not others) so we should probably have a way to add ftz behaviors a la carte.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">FWIW, ICC sets the FTZ and DAZ flags from a function call that is inserted into main depending on the options used to compile the file containing main.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Trying to go back to the general case, I'd like to solicit information about whether other targets have/need different denormal options than are described above. Futher, I'd suggest that for any architecture that supports FTZ behavior,
a well-document default be automatically set when fast math is enabled via<o:p></o:p></p>
<p class="MsoNormal">-Ofast, -ffast-math, or -funsafe-math-optimizations unless that option is turned off by a subsequent -fno-fast-math/-fno-unsafe-math-optimizations option or overridden by a subsequent -fdenormal-fp-math option, and if -fdenormal-fp-math
is used, some code will be emitted to set the relevant hardware controls.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I don't have a strong opinion on whether it is better to emit a static constructor or to inject a call into main. The latter seems more predictable. I’d like to avoid a dependency on crtfastmath.o either way.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Do we need an ftz fast-math flag?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Are there any other facets to this problem that I've overlooked?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Andy<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>