[flang-commits] [flang] 4a3e4b9 - [flang] Adjust prescanner fix for preprocessing (#122779)

[via flang-commits]

Author: Peter Klausler
Date: 2025-01-14T13:01:31-08:00
New Revision: 4a3e4b99b9ab3016afe8b02c4f83f24635964f4e

URL: https://github.com/llvm/llvm-project/commit/4a3e4b99b9ab3016afe8b02c4f83f24635964f4e
DIFF: https://github.com/llvm/llvm-project/commit/4a3e4b99b9ab3016afe8b02c4f83f24635964f4e.diff

LOG: [flang] Adjust prescanner fix for preprocessing (#122779)

Commas being optional in FORMAT statements, the tokenization of things
like 3I9HHOLLERITH is tricky. After tokenizing the initial '3', we don't
want to then take apparent identifier "I9HHOLLERITH" as the next token.
So the prescanner just consumes the letter ("I") as its own token in
this context.

A recent bug report complained that this can lead to incorrect results
when (in this case) the letter is a defined preprocessing macro. I
updated the prescanner to check that the letter is actually followed by
an instance of a problematic Hollerith literal.

And this broke two tests in the Fujitsu Fortran test suite that Linaro
runs, as it couldn't detect a following Hollerith literal that wasn't on
the same source line. We can't do look-ahead line continuation
processing in NextToken(), either.

So here's a second attempt at fixing the original problem: namely, the
letter that follows a decimal integer token is checked to see whether
it's the name of a defined macro.

Added: 
    

Modified: 
    flang/lib/Parser/prescan.cpp
    flang/test/Preprocessing/bug129131.F

Removed: 
    


################################################################################
diff  --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 703a02792a1c4e..c5939a1e0b6c2c 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -708,23 +708,11 @@ bool Prescanner::NextToken(TokenSequence &tokens) {
       EmitCharAndAdvance(tokens, *at_);
       QuotedCharacterLiteral(tokens, start);
     } else if (IsLetter(*at_) && !preventHollerith_ &&
-        parenthesisNesting_ > 0) {
-      const char *p{at_};
-      int digits{0};
-      for (;; ++digits) {
-        ++p;
-        if (InFixedFormSource()) {
-          p = SkipWhiteSpace(p);
-        }
-        if (!IsDecimalDigit(*p)) {
-          break;
-        }
-      }
-      if (digits > 0 && (*p == 'h' || *p == 'H')) {
-        // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
-        // we don't misrecognize I9HOLLERITH as an identifier in the next case.
-        EmitCharAndAdvance(tokens, *at_);
-      }
+        parenthesisNesting_ > 0 &&
+        !preprocessor_.IsNameDefined(CharBlock{at_, 1})) {
+      // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
+      // we don't misrecognize I9HHOLLERITH as an identifier in the next case.
+      EmitCharAndAdvance(tokens, *at_);
     }
     preventHollerith_ = false;
   } else if (*at_ == '.') {

diff  --git a/flang/test/Preprocessing/bug129131.F b/flang/test/Preprocessing/bug129131.F
index 5b1a914a2c9e35..00aba5da2c7cbd 100644
--- a/flang/test/Preprocessing/bug129131.F
+++ b/flang/test/Preprocessing/bug129131.F
@@ -1,5 +1,8 @@
 ! RUN: %flang -fc1 -fdebug-unparse %s 2>&1 | FileCheck %s
 ! CHECK: PRINT *, 2_4
+! CHECK: PRINT *, 1_4
 #define a ,3
       print *, mod(5 a)
+      print *, mod(4 a
+     +)
       end