[flang-commits] [flang] 776e25a - [flang] Inhibit case of false tokenization of Hollerith (#79029)

[via flang-commits]

Author: Peter Klausler
Date: 2024-01-25T16:09:17-08:00
New Revision: 776e25af6d927bc6be026070181c834069fbf742

URL: https://github.com/llvm/llvm-project/commit/776e25af6d927bc6be026070181c834069fbf742
DIFF: https://github.com/llvm/llvm-project/commit/776e25af6d927bc6be026070181c834069fbf742.diff

LOG: [flang] Inhibit case of false tokenization of Hollerith (#79029)

https://github.com/llvm/llvm-project/issues/78927 contains a case of
fixed-form source in which a Hollerith literal is mistakenly tokenized,
leading to grief later due to apparently unbalanced parentheses.

The source looks like "REAL*8 R8HEAP(SCRSIZE)" and the Hollerith literal
is misrecognized as such because it follows "8R". In order to properly
tokenize Hollerith literals in old comma-free FORMAT statements like "1
FORMAT(3I5HFLANG)", the tokenizer in the prescanner treats a letter
after an integer token ("3I") as a special case. The fix is to do this
only when the characters involved are nested in parentheses and
Hollerith is a possibility.

Fixes https://github.com/llvm/llvm-project/issues/78927.

Added: 
    

Modified: 
    flang/lib/Parser/prescan.cpp

Removed: 
    


################################################################################
diff  --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 6bccfc3f9baea9f..f7f22177a7d0bfa 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -608,13 +608,14 @@ bool Prescanner::NextToken(TokenSequence &tokens) {
       do {
         EmitCharAndAdvance(tokens, *at_);
       } while (IsHexadecimalDigit(*at_));
-    } else if (IsLetter(*at_)) {
-      // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
-      // we don't misrecognize I9HOLLERITH as an identifier in the next case.
-      EmitCharAndAdvance(tokens, *at_);
     } else if (at_[0] == '_' && (at_[1] == '\'' || at_[1] == '"')) { // 4_"..."
       EmitCharAndAdvance(tokens, *at_);
       QuotedCharacterLiteral(tokens, start);
+    } else if (IsLetter(*at_) && !preventHollerith_ &&
+        parenthesisNesting_ > 0) {
+      // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
+      // we don't misrecognize I9HOLLERITH as an identifier in the next case.
+      EmitCharAndAdvance(tokens, *at_);
     }
     preventHollerith_ = false;
   } else if (*at_ == '.') {