[flang-commits] [flang] [flang][runtime] Fix integer overflow check in FORMAT (PR #79368)

[Peter Klausler via flang-commits]

https://github.com/klausler created https://github.com/llvm/llvm-project/pull/79368

The integer overflow check used for repeat counts &c. in FORMAT specifications was incorrect; fix it.

Fixes https://github.com/llvm/llvm-project/issues/79255.

>From c1d8138cb38f80690f67ec47ade65c86af5fca24 Mon Sep 17 00:00:00 2001
From: Peter Klausler <pklausler at nvidia.com>
Date: Wed, 24 Jan 2024 12:52:25 -0800
Subject: [PATCH] [flang][runtime] Fix integer overflow check in FORMAT

The integer overflow check used for repeat counts &c. in FORMAT
specifications was incorrect; fix it.

Fixes https://github.com/llvm/llvm-project/issues/79255.
---
 flang/runtime/format-implementation.h | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/flang/runtime/format-implementation.h b/flang/runtime/format-implementation.h
index c54ac062c7beab9..e475508b9ba58a7 100644
--- a/flang/runtime/format-implementation.h
+++ b/flang/runtime/format-implementation.h
@@ -85,8 +85,9 @@ int FormatControl<CONTEXT>::GetIntField(
     ch = PeekNext();
   }
   while (ch >= '0' && ch <= '9') {
-    if (result >
-        std::numeric_limits<int>::max() / 10 - (static_cast<int>(ch) - '0')) {
+    if (result > std::numeric_limits<int>::max() / 10 ||
+        std::numeric_limits<int>::max() - 10 * result <
+            static_cast<int>(ch) - '0') {
       handler.SignalError(
           IostatErrorInFormat, "FORMAT integer field out of range");
       if (hadError) {