[flang-commits] [flang] [flang] Inhibit case of false tokenization of Hollerith (PR #79029)

[via flang-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-flang-parser

Author: Peter Klausler (klausler)

<details>
<summary>Changes</summary>

https://github.com/llvm/llvm-project/issues/78927 contains a case of fixed-form source in which a Hollerith literal is mistakenly tokenized, leading to grief later due to apparently unbalanced parentheses.

The source looks like "REAL*8 R8HEAP(SCRSIZE)" and the Hollerith literal is misrecognized as such because it follows "8R".  In order to properly tokenize Hollerith literals in old comma-free FORMAT statements like "1 FORMAT(3I5HFLANG)", the tokenizer in the prescanner treats a letter after an integer token ("3I") as a special case. The fix is to do this only when the characters involved are nested in parentheses and Hollerith is a possibility.

Fixes https://github.com/llvm/llvm-project/issues/78927.

---
Full diff: https://github.com/llvm/llvm-project/pull/79029.diff


1 Files Affected:

- (modified) flang/lib/Parser/prescan.cpp (+5-4) 


``````````diff
diff --git a/flang/lib/Parser/prescan.cpp b/flang/lib/Parser/prescan.cpp
index 68d7d9f0c53c475..029652adbca1df5 100644
--- a/flang/lib/Parser/prescan.cpp
+++ b/flang/lib/Parser/prescan.cpp
@@ -605,13 +605,14 @@ bool Prescanner::NextToken(TokenSequence &tokens) {
       do {
         EmitCharAndAdvance(tokens, *at_);
       } while (IsHexadecimalDigit(*at_));
-    } else if (IsLetter(*at_)) {
-      // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
-      // we don't misrecognize I9HOLLERITH as an identifier in the next case.
-      EmitCharAndAdvance(tokens, *at_);
     } else if (at_[0] == '_' && (at_[1] == '\'' || at_[1] == '"')) { // 4_"..."
       EmitCharAndAdvance(tokens, *at_);
       QuotedCharacterLiteral(tokens, start);
+    } else if (IsLetter(*at_) && !preventHollerith_ &&
+        parenthesisNesting_ > 0) {
+      // Handles FORMAT(3I9HHOLLERITH) by skipping over the first I so that
+      // we don't misrecognize I9HOLLERITH as an identifier in the next case.
+      EmitCharAndAdvance(tokens, *at_);
     }
     preventHollerith_ = false;
   } else if (*at_ == '.') {

``````````

</details>


https://github.com/llvm/llvm-project/pull/79029