[flang-commits] [flang] [flang] Don't create bogus tokens from token pasting (##) (PR #147596)

[Eugene Epshteyn via flang-commits]

================
@@ -156,23 +156,50 @@ static TokenSequence TokenPasting(TokenSequence &&text) {
   }
   TokenSequence result;
   std::size_t tokens{text.SizeInTokens()};
-  bool pasting{false};
+  std::optional<CharBlock> before; // last non-blank token before ##
   for (std::size_t j{0}; j < tokens; ++j) {
-    if (IsTokenPasting(text.TokenAt(j))) {
-      if (!pasting) {
+    CharBlock after{text.TokenAt(j)};
+    if (!before) {
+      if (IsTokenPasting(after)) {
         while (!result.empty() &&
             result.TokenAt(result.SizeInTokens() - 1).IsBlank()) {
           result.pop_back();
         }
         if (!result.empty()) {
-          result.ReopenLastToken();
-          pasting = true;
+          before = result.TokenAt(result.SizeInTokens() - 1);
         }
+      } else {
+        result.AppendRange(text, j, 1);
+      }
+    } else if (after.IsBlank() || IsTokenPasting(after)) {
+      // drop it
+    } else { // pasting before ## after
+      bool doPaste{false};
+      char last{before->end()[-1]};
+      char first{after.begin()[0]};
----------------
eugeneepshteyn wrote:

Just a suggestion: maybe there could be `char CharBlock::first()` and `char CharBlock::last()`. There are a few instances of  'begin()[0]` and `end()[-1]` in the code that could then be replaced by `first()` and `last()`.

https://github.com/llvm/llvm-project/pull/147596