[clang] [clang][Tooling] Ignore end-of-directive tokens in TokenCollector (PR #212242)

[Macro Terra via cfe-commits]

https://github.com/hongtaihu created https://github.com/llvm/llvm-project/pull/212242

### Summary

`TokenCollector` receives `tok::eod` from the preprocessor for an end-of-directive marker, but that token has no one-to-one raw spelling. Recording it as an expanded token can therefore make the expanded-to-spelled mapping fail in clangd.

Skip `tok::eod` alongside non-module annotation tokens. Keep the existing `annot_module_name` reconstruction path, as it intentionally supplies a spelling.

Add a regression test for:

```cpp
}
#pragma clang __debug dump
```

### Test Plan

- `tools/clang/unittests/AllClangUnitTests --gtest_filter=TokenCollectorTest.DebugPragmaAtEndOfFile`
- Replayed issue #197652 through clangd `textDocument/hover`; clangd returned the hover response and no longer hit the expanded-to-spelled mapping assertion.


>From f00c6734cc3b015166cb80309ad6cc975a2de2da Mon Sep 17 00:00:00 2001
From: hongtaihu <13541229370 at qq.com>
Date: Mon, 27 Jul 2026 20:58:49 +0800
Subject: [PATCH] [clang][Tooling] Ignore end-of-directive tokens in
 TokenCollector

---
 clang/lib/Tooling/Syntax/Tokens.cpp           | 9 ++++++---
 clang/unittests/Tooling/Syntax/TokensTest.cpp | 9 +++++++++
 2 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/clang/lib/Tooling/Syntax/Tokens.cpp b/clang/lib/Tooling/Syntax/Tokens.cpp
index 927333fda18bb..6b75c17cab67e 100644
--- a/clang/lib/Tooling/Syntax/Tokens.cpp
+++ b/clang/lib/Tooling/Syntax/Tokens.cpp
@@ -688,11 +688,14 @@ TokenCollector::TokenCollector(Preprocessor &PP) : PP(PP) {
           this->PP.getLangOpts());
       Expanded.push_back(
           syntax::Token(T.getLocation(), Text.size(), tok::annot_module_name));
-    } else if (T.isAnnotation()) {
       return;
-    } else {
-      Expanded.push_back(syntax::Token(T));
     }
+
+    // These tokens do not have a one-to-one raw spelling.
+    if (T.isAnnotation() || T.is(tok::eod))
+      return;
+
+    Expanded.push_back(syntax::Token(T));
     DEBUG_WITH_TYPE("collect-tokens", llvm::dbgs()
                                           << "Token: "
                                           << syntax::Token(T).dumpForTests(
diff --git a/clang/unittests/Tooling/Syntax/TokensTest.cpp b/clang/unittests/Tooling/Syntax/TokensTest.cpp
index 6418cc8f87d9d..ebc640b8d0923 100644
--- a/clang/unittests/Tooling/Syntax/TokensTest.cpp
+++ b/clang/unittests/Tooling/Syntax/TokensTest.cpp
@@ -1169,6 +1169,15 @@ TEST_F(TokenCollectorTest, Pragmas) {
   )cpp");
 }
 
+TEST_F(TokenCollectorTest, DebugPragmaAtEndOfFile) {
+  AllowErrors = true;
+  recordTokens("}\n#pragma clang __debug dump\n");
+
+  // The end-of-directive token has no spelling and must not be collected.
+  EXPECT_THAT(Buffer.expandedTokens(),
+              ElementsAre(Kind(tok::r_brace), Kind(tok::eof)));
+}
+
 TEST_F(TokenBufferTest, EofTokenOnBracketDepthLimit) {
   // Force parser to bail out due to exceeding the bracket depth limit.
   recordTokens("((;", {"-fbracket-depth=1"});