[flang-commits] [flang] [flang][runtime] Fix off-by-one error in EX0.0 output editing (PR #85428)

[Peter Klausler via flang-commits]

https://github.com/klausler created https://github.com/llvm/llvm-project/pull/85428

The maximum number of significant hexadecimal digits in EX0.0 REAL output editing is 29, not 28.  Fix by computing it at build time from the precision of REAL(16).

>From 5bbb67e97bc64f954f9e5331f5ba0afcabe7b272 Mon Sep 17 00:00:00 2001
From: Peter Klausler <pklausler at nvidia.com>
Date: Fri, 15 Mar 2024 09:47:08 -0700
Subject: [PATCH] [flang][runtime] Fix off-by-one error in EX0.0 output editing

The maximum number of significant hexadecimal digits in EX0.0
REAL output editing is 29, not 28.  Fix by computing it at
build time from the precision of REAL(16).
---
 flang/runtime/edit-output.cpp | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/flang/runtime/edit-output.cpp b/flang/runtime/edit-output.cpp
index b474c8cd91bae4..7267540370fc07 100644
--- a/flang/runtime/edit-output.cpp
+++ b/flang/runtime/edit-output.cpp
@@ -9,6 +9,7 @@
 #include "edit-output.h"
 #include "emit-encoded.h"
 #include "utf.h"
+#include "flang/Common/real.h"
 #include "flang/Common/uint128.h"
 #include <algorithm>
 
@@ -700,7 +701,9 @@ bool RealOutputEditing<KIND>::EditEXOutput(const DataEdit &edit) {
   if ((editWidth == 0 && !edit.digits) || editDigits == 0) {
     // EX0 or EXw.0
     flags |= decimal::Minimize;
-    significantDigits = 28; // enough for 128-bit F.P.
+    static constexpr int maxSigHexDigits{
+        (common::PrecisionOfRealKind(16) + 3) / 4};
+    significantDigits = maxSigHexDigits;
   }
   auto converted{
       ConvertToHexadecimal(significantDigits, edit.modes.round, flags)};