[flang-commits] [flang] [Flang][Preprocessor] Avoid creating an empty token when a kind suffix is torn by a pasting operator (PR #139795)

[Roger Ferrer Ibáñez via flang-commits]

https://github.com/rofirrim created https://github.com/llvm/llvm-project/pull/139795

This input "tears" the expected tokens of an integer-literal due to a pasting operator `##`. When lexing `1_##` we generate the sequence of tokens `['1_', '']`, the second being an empty token of length zero. The second token is created at the end of `Prescanner::NextToken`.

Creating an empty token by accident (due to two consecutive `CloseToken` without consuming anything) can cause `TokenSequence::pop_back` to assert.

If zero-length tokens are acceptable, then instead of this patch we may have to fix the logic in `TokenPasting` found in `preprocessor.cpp`.

>From 4e1f840899384d9977105791785806b2c43c1a37 Mon Sep 17 00:00:00 2001
From: Roger Ferrer Ibanez <roger.ferrer at bsc.es>
Date: Tue, 13 May 2025 20:54:14 +0000
Subject: [PATCH] Avoid creating an empty token when a kind suffix is torn by a
 pasting operator

This confuses the logic that implements the pasting itself.
---
 flang/lib/Parser/prescan.cpp                      | 8 ++++++++
 flang/test/Preprocessing/torn-token-pasting-1.F90 | 9 +++++++++
 2 files changed, 17 insertions(+)
 create mode 100644 flang/test/Preprocessing/torn-token-pasting-1.F90

diff --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 3bc2ea0b37508..004e4f013f90a 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -937,6 +937,7 @@ bool Prescanner::HandleKindSuffix(TokenSequence &tokens) {
   if (*at_ != '_') {
     return false;
   }
+  auto underscore = *at_;
   TokenSequence withUnderscore, separate;
   EmitChar(withUnderscore, '_');
   EmitCharAndAdvance(separate, '_');
@@ -951,6 +952,13 @@ bool Prescanner::HandleKindSuffix(TokenSequence &tokens) {
   }
   withUnderscore.CloseToken();
   separate.CloseToken();
+  // If we only saw "_" and nothing else, we have handled enough but we do not
+  // want to close the token here, or we will generate an extra token of length
+  // zero.
+  if (separate.SizeInTokens() == 1) {
+    EmitChar(tokens, underscore);
+    return true;
+  }
   tokens.CloseToken();
   if (separate.SizeInTokens() == 2 &&
       preprocessor_.IsNameDefined(separate.TokenAt(1)) &&
diff --git a/flang/test/Preprocessing/torn-token-pasting-1.F90 b/flang/test/Preprocessing/torn-token-pasting-1.F90
new file mode 100644
index 0000000000000..5e080129a94d1
--- /dev/null
+++ b/flang/test/Preprocessing/torn-token-pasting-1.F90
@@ -0,0 +1,9 @@
+! RUN: %flang -E %s 2>&1 | FileCheck %s
+! CHECK: IF(10>HUGE(1_4).OR.10<-HUGE(1_4)) CALL foo()
+#define CHECKSAFEINT(x,k)  IF(x>HUGE(1_  ##  k).OR.x<-HUGE(1_##k)) CALL foo()
+
+program main
+  implicit none
+
+  CHECKSAFEINT(10, 4)
+end program main