[PATCH] D143074: [LangRef] improve documentation of SNaN in the default FP environment

Sat Feb 4 14:34:12 PST 2023

jyknight added a comment.

In D143074#4104239 <https://reviews.llvm.org/D143074#4104239>, @RalfJung wrote:

>> Floating-point math operations assume that all NaNs are quiet.
>
> That could be read as "if you produce a signaling NaN (e.g. via a bitcast) and feed that to a math operation, that violates the assumption, and hence is UB".

I'm happy to have as little "undefinedness" as possible...but...

> I think it'd be better to explicitly say that when a math operation produces a NaN, it non-deterministically picks an arbitrary (quiet or signaling) NaN.

What I was trying to state is that this is insufficient (but failed to do so clearly, sorry). Yet, for basically the same reason, it's also too permissive: we cannot allow LLVM to spuriously introduce sNaNs when the original code did not use any.

As a general rule, when an operation gets an sNaN as input, it raises an invalid exception immediately. The default behavior upon is to set the invalid bit in the status flags AND to trigger an immediate return of a qNaN -- even when a qNaN input value would've resulted in some other output.

Taking `pow` in particular, the correct results are:

  pow(1.0, sNaN) -> qNaN
  pow(1.0, qNaN) -> 1.0
  pow(sNaN, 0) -> qNaN
  pow(qNaN, 0) -> 1.0

If we canonicalize `pow(1.0, <anything>) -> 1.0` and `pow(<anything>, 0) -> 1.0`, then `pow(sNaN, 0) -> 1.0`, instead of `qNaN`.

Of course, we may also get the wrong answer in the other direction: `pow(1.0 * x, y)` should result in 1.0 when passed `x = sNaN`, `y = 0` (because `1.0 * sNaN -> qNaN`, and `pow(qNaN, 0) -> 1.0`); but if we canonicalize away the multiplication to `pow(x, y)`, we may up with qNaN as the result.

Along the same lines, if we were to allow any operation to return sNaN instead of qNaN -- even when no sNaN has been provided as input -- then we'd allow something like `pow(2.0 * qNaN, 0.0)` to non-deterministically result in qNaN or 1.0, which is not OK.

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D143074/new/

https://reviews.llvm.org/D143074