[PATCH] D68257: [Support] Add mathematical constants

[Evandro Menezes via Phabricator via llvm-commits]

evandro marked 3 inline comments as done.
evandro added inline comments.


================
Comment at: llvm/include/llvm/Support/MathExtras.h:66
+                 inv_sqrtpi = 0.5641895835477563, // https://oeis.org/A087197
+                 sqrt2      = 1.414213562373095, // https://oeis.org/A002193
+                 inv_sqrt2  = 0.7071067811865475,
----------------
efriedma wrote:
> evandro wrote:
> > efriedma wrote:
> > > The correct value of sqrt(2) in double-precision is 1.4142135623730951.
> > > 
> > > And now I don't trust any of the other values...
> > `double` has a precision of 15 or 16 significant digits.  I don't understand why are you suggesting 17 significant digits when you asked to trim the precision down.
> > 
> > Besides, the reference I provided states that this value is 1.41421356237309505.  Whether it's rounded to 1.4142135623730950 or 1.4142135623730951 is a bit moot, IMO.
> I asked for "the smallest number of digits required to produce the correct double-precision result".  This is what you get if, for example, you ask Python 2.7 or later to convert the value to a string with `repr()` (`printf "import math\nprint(repr(math.sqrt(2)))" | python`).  `1.414213562373095` produces a value that's different by one ulp.
> 
> Yes, a one ulp difference is unlikely to matter for most uses, but if we're going to take the time to define these, we should define them correctly.
You're assuming that Python is correct.  `bc` says 1.41421356237309504880.  glibc's `math.h` says 1.41421356237309504880 as well.  And none of these is the same as your 1.4142135623730951.  

As I said, the precision of `double` is 15 to 16 digits and of `float`, 6 to 7 digits.  `math.h` defines them with 20 digits, which is probably an agreeable precision, yes?  But I believe that we call all live with a difference of ±1ulp.


CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D68257/new/

https://reviews.llvm.org/D68257