[clang] [Syntax] Append EOF token to truncated expanded token stream when the parser halts prematurely (PR #196861)

[Haojian Wu via cfe-commits]

================
@@ -712,6 +712,16 @@ class TokenCollector::Builder {
 
   TokenBuffer build() && {
     assert(!Result.ExpandedTokens.empty());
+
+    // When the parser hits a hard limit (e.g. bracket depth or function scope
+    // depth), it halts prematurely and leaves the expanded token stream
+    // truncated with no final `eof` token. To keep the invariant, synthesize an
+    // `eof` at the location of the last collected token.
+    if (Result.ExpandedTokens.back().kind() != tok::eof) {
+      SourceLocation Loc = Result.ExpandedTokens.back().location();
+      Result.ExpandedTokens.emplace_back(Loc, 0, tok::eof);
+    }
+
     assert(Result.ExpandedTokens.back().kind() == tok::eof);
----------------
hokein wrote:

we can remove this assert now.

https://github.com/llvm/llvm-project/pull/196861