[clang] [clang][Tooling] Ignore end-of-directive tokens in TokenCollector (PR #212242)

[via cfe-commits]

llvmorg-github-actions[bot] wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-clang

Author: Macro Terra (hongtaihu)

<details>
<summary>Changes</summary>

fixes #<!-- -->197652
### Summary

`TokenCollector` receives `tok::eod` from the preprocessor for an end-of-directive marker, but that token has no one-to-one raw spelling. Recording it as an expanded token can therefore make the expanded-to-spelled mapping fail in clangd.

Skip `tok::eod` alongside non-module annotation tokens. Keep the existing `annot_module_name` reconstruction path, as it intentionally supplies a spelling.

Add a regression test for:

```cpp
}
#pragma clang __debug dump
```

assisted by codex.


---
Full diff: https://github.com/llvm/llvm-project/pull/212242.diff


2 Files Affected:

- (modified) clang/lib/Tooling/Syntax/Tokens.cpp (+6-3) 
- (modified) clang/unittests/Tooling/Syntax/TokensTest.cpp (+9) 


``````````diff
diff --git a/clang/lib/Tooling/Syntax/Tokens.cpp b/clang/lib/Tooling/Syntax/Tokens.cpp
index 927333fda18bb..6b75c17cab67e 100644
--- a/clang/lib/Tooling/Syntax/Tokens.cpp
+++ b/clang/lib/Tooling/Syntax/Tokens.cpp
@@ -688,11 +688,14 @@ TokenCollector::TokenCollector(Preprocessor &PP) : PP(PP) {
           this->PP.getLangOpts());
       Expanded.push_back(
           syntax::Token(T.getLocation(), Text.size(), tok::annot_module_name));
-    } else if (T.isAnnotation()) {
       return;
-    } else {
-      Expanded.push_back(syntax::Token(T));
     }
+
+    // These tokens do not have a one-to-one raw spelling.
+    if (T.isAnnotation() || T.is(tok::eod))
+      return;
+
+    Expanded.push_back(syntax::Token(T));
     DEBUG_WITH_TYPE("collect-tokens", llvm::dbgs()
                                           << "Token: "
                                           << syntax::Token(T).dumpForTests(
diff --git a/clang/unittests/Tooling/Syntax/TokensTest.cpp b/clang/unittests/Tooling/Syntax/TokensTest.cpp
index 6418cc8f87d9d..ebc640b8d0923 100644
--- a/clang/unittests/Tooling/Syntax/TokensTest.cpp
+++ b/clang/unittests/Tooling/Syntax/TokensTest.cpp
@@ -1169,6 +1169,15 @@ TEST_F(TokenCollectorTest, Pragmas) {
   )cpp");
 }
 
+TEST_F(TokenCollectorTest, DebugPragmaAtEndOfFile) {
+  AllowErrors = true;
+  recordTokens("}\n#pragma clang __debug dump\n");
+
+  // The end-of-directive token has no spelling and must not be collected.
+  EXPECT_THAT(Buffer.expandedTokens(),
+              ElementsAre(Kind(tok::r_brace), Kind(tok::eof)));
+}
+
 TEST_F(TokenBufferTest, EofTokenOnBracketDepthLimit) {
   // Force parser to bail out due to exceeding the bracket depth limit.
   recordTokens("((;", {"-fbracket-depth=1"});

``````````

</details>


https://github.com/llvm/llvm-project/pull/212242