[llvm-bugs] [Bug 31294] New: llvm::APFloat when using x87DoubleExtended semantics does not handle unsupported operands properly

[via llvm-bugs]

https://llvm.org/bugs/show_bug.cgi?id=31294

            Bug ID: 31294
           Summary: llvm::APFloat when using x87DoubleExtended semantics
                    does not handle unsupported operands properly
           Product: libraries
           Version: trunk
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P
         Component: Support Libraries
          Assignee: unassignedbugs at nondot.org
          Reporter: dan at su-root.co.uk
                CC: andrew.kaylor at intel.com, david.l.kreitzer at intel.com,
                    geek4civic at gmail.com, hfinkel at anl.gov,
                    llvm-bugs at lists.llvm.org, scanon at apple.com
    Classification: Unclassified

Created attachment 17728
  --> https://llvm.org/bugs/attachment.cgi?id=17728&action=edit
Program testing native execution vs llvm::APFloat

In the Intel 64 and IA-32 Architectures Software Developer's Manual in section
8.2.2 "Unsupported Double Extended-Precision Floating-Point Encodings and
Pseudo-Denormals" several classes of floating point number are defined.

These are positive and negative versions of

* Pseudo-NaNs Quiet
* Pseudo-NaNs Signaling
* Pseudo-infinity
* Unnormals
* Pseudo-denormals

All of these except (Pseudo-denormals) are considered to be invalid operands by
the manual mentioned above.

I hacked up some code (attached) that can be used to compare the native
behaviour vs what APFloat does. I only had time to examine the positive
versions of the classes above. It's likely that the negative versions have
similar problems.

In attached source comments with `BUG` in them are places where I think there
might be a bug. Here's a summary of problems I've found so far

To compile run

```
clang++ x87_fp80_vs_APFloat.cpp -Wall $(llvm-config --cxxflags) -L$(llvm-config
--libdir) $(llvm-config --libs Support) $(llvm-config --system-libs Support)
-O0

```

Note the `-O0` is very important. Clang seems to be constant folding at `-O2`
which changes the behaviour of the program and we inherit `-O2` from
`llvm-config --cxxflags`.

# Classes

## Positive Pseudo-NaNs quiet

- isnanl() returns true for this value. I'm not sure if that's a bug or not.
This is a glibc issue though so it's just a passing observation and not a
problem with LLVM.
- APFloat::isNaN() returns true. I'm not sure if it should as these are not
IEEE754 NaNs
- When performing addition on this value native and APFloat differ. Native
raises a invalid operation exception (as expected) but APFloat does not and
returns a different bit pattern. Native returns a -ve quiet NaN but APFloat
returns the value (i.e positive quiet pseudo-nan)

## Positive Pseudo-NaN signalling

- isnanl() returns true for this value. I'm not sure if that's a bug or not.
This is a glibc issue though so it's just a passing observation and not a
problem with LLVM.
- APFloat::isNaN() and APFloat::isSignaling() returns true. I'm not sure if it
should as these are not IEEE754 NaNs
- When performing addition on this value native and APFloat differ. Native
raises a invalid operation exception (as expected) but APFloat does not and
returns a different bit pattern. Native returns a -ve quiet NaN but APFloat
returns the value (i.e positive signaling pseudo-nan)

## Positive pseudo-infinity

- `isnanl()` returns true for this value and `isinfl()` returns 0. I'm not sure
if that's a bug or not. This is a glibc issue though so it's just a passing
observation and not a problem with LLVM.
- APFloat::isInfinity() returns false and APFloat::IsNaN() returns true. I'm
not sure what the right result should be here.
- When performing addition on this value native and APFloat differ. Native
raises an invalid operation exception (as expected) but APFloat does not and
returns a different bit pattern. Native returns a -ve quiet NaN but APFloat
returns the value (i.e positive pseudo-infinity)

## Positive Unnormals

- When comparing this value against itself (`!=`) the native and APFloat
results differ. Natively this raises an invalid operation exception and returns
false for the comparision. APFloat returns true and returns the Inexact
exception.
- When performing addition on this value native and APFloat differ. Native
raises an invalid operation exception (as expected) but APFloat does not and
returns a different bit pattern.  Native returns a -ve quiet NaN but APFloat
returns a normal floating point number.

It is possible to use this mishandling of unnormals by APFloat to trigger an
assertion failure in upstream Clang when building with `-O2`. I've reported
this separately

https://llvm.org/bugs/show_bug.cgi?id=31292

## Positive pseudo denormals

Native and APFloat agree which surprised me. The Intel manual is a bit unclear
here because it says that starting with 387 math coprocessor operands of this
sort are not generated by IA-32 processors but when these appear as operands
they are handled "correctly". It says that IA-32 processors support this for
legacy code. It doesn't say what happens with x86_64 but this single example
suggests that it is handled in the same way.

# A few high level thoughts

* Fixing this is potentially problematic because how these operands are treated
depends on the version of Intel hardware you are using. In most cases though I
would expect very few Intel cpus older than the 386 to be in use (Can LLVM even
codegen for this?) so the simplest is solution is to follow the semantics used
by the 386 and newer. If we wanted to support the old CPUs we'd need to support
two versions of the `x87DoubleExtended` semantics and use the appropriate
version depending on the target.

* I'm not sure what the `isNaN()`, `isInfinity()` (and others) should return
for these unsupported values. It would probably be nice to have a predicate
that could identify these unsupported values.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20161206/77f14d00/attachment-0001.html>