[flang-commits] [flang] [Flang][Preprocessor] Avoid creating an empty token when a kind suffix is torn by a pasting operator (PR #139795)

[via flang-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-flang-parser

Author: Roger Ferrer Ibáñez (rofirrim)

<details>
<summary>Changes</summary>

This input "tears" the expected tokens of an integer-literal due to a pasting operator `##`. When lexing `1_##` we generate the sequence of tokens `['1_', '']`, the second being an empty token of length zero. The second token is created at the end of `Prescanner::NextToken`.

Creating an empty token by accident (due to two consecutive `CloseToken` without consuming anything) can cause `TokenSequence::pop_back` to assert.

If zero-length tokens are acceptable, then instead of this patch we may have to fix the logic in `TokenPasting` found in `preprocessor.cpp`.

---
Full diff: https://github.com/llvm/llvm-project/pull/139795.diff


2 Files Affected:

- (modified) flang/lib/Parser/prescan.cpp (+8) 
- (added) flang/test/Preprocessing/torn-token-pasting-1.F90 (+9) 


``````````diff
diff --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 3bc2ea0b37508..004e4f013f90a 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -937,6 +937,7 @@ bool Prescanner::HandleKindSuffix(TokenSequence &tokens) {
   if (*at_ != '_') {
     return false;
   }
+  auto underscore = *at_;
   TokenSequence withUnderscore, separate;
   EmitChar(withUnderscore, '_');
   EmitCharAndAdvance(separate, '_');
@@ -951,6 +952,13 @@ bool Prescanner::HandleKindSuffix(TokenSequence &tokens) {
   }
   withUnderscore.CloseToken();
   separate.CloseToken();
+  // If we only saw "_" and nothing else, we have handled enough but we do not
+  // want to close the token here, or we will generate an extra token of length
+  // zero.
+  if (separate.SizeInTokens() == 1) {
+    EmitChar(tokens, underscore);
+    return true;
+  }
   tokens.CloseToken();
   if (separate.SizeInTokens() == 2 &&
       preprocessor_.IsNameDefined(separate.TokenAt(1)) &&
diff --git a/flang/test/Preprocessing/torn-token-pasting-1.F90 b/flang/test/Preprocessing/torn-token-pasting-1.F90
new file mode 100644
index 0000000000000..5e080129a94d1
--- /dev/null
+++ b/flang/test/Preprocessing/torn-token-pasting-1.F90
@@ -0,0 +1,9 @@
+! RUN: %flang -E %s 2>&1 | FileCheck %s
+! CHECK: IF(10>HUGE(1_4).OR.10<-HUGE(1_4)) CALL foo()
+#define CHECKSAFEINT(x,k)  IF(x>HUGE(1_  ##  k).OR.x<-HUGE(1_##k)) CALL foo()
+
+program main
+  implicit none
+
+  CHECKSAFEINT(10, 4)
+end program main

``````````

</details>


https://github.com/llvm/llvm-project/pull/139795