[clang] [clang] Bounds checking on unclosed parentheses, brackets or braces in Expanded Tokens (PR #69849)

[Nathan Ridge via cfe-commits]

HighCommander4 wrote:

I poked at this a bit more. A few more notes:

 * Since the test case currently in the patch does not trigger the crash (without the fix), and it was not obvious to me what sort of call to `spelledForExpanded()` to formulate to trigger the crash, I instead formulated a test case in `DumpASTTests.cpp` that calls `dumpAST()` on the `TranslationUnitDecl` of a file containing the code `int main() {`. This does trigger the crash.
 * The crash occurs when dumping the `FunctionDecl` node for `main`. When dumping a node, `dumpAST()` [calls](https://searchfox.org/llvm/rev/fd1c156e5d6584684ce58c0536dca96cedcc41f0/clang-tools-extra/clangd/DumpAST.cpp#93) `Tokens.spelledForExpanded(Tokens.expandedTokens(SR))` where `SR` is the node's source range.
 * For this `FunctionDecl` node, the end location of its source range points to a `tok::eof` token.
    * This comes from the end location of the function's `CompoundStmt` body, which is what the parser assigns in [Parser::ParseCompoundStatementBody()](https://searchfox.org/llvm/rev/fd1c156e5d6584684ce58c0536dca96cedcc41f0/clang/lib/Parse/ParseStmt.cpp#1292) in this situation.
 * The range of expanded tokens returned by `Tok.expandedTokens(SR)` thus includes as its last element the `tok::eof` token.
 * The [comment](https://searchfox.org/llvm/rev/fd1c156e5d6584684ce58c0536dca96cedcc41f0/clang/include/clang/Tooling/Syntax/Tokens.h#162) above `TokenBuffer` says "the expanded token stream has a `tok::eof` token at the end, the spelled tokens never store a 'eof' token".

I haven't dug into the implementation of `spelledForExpanded()` yet, but from the above it's looking to me like `spelledForExpanded()` is not expecting an input expanded token range that includes the `eof` token, but we are passing it one (and such a token range can legitimately arise from the source range of the AST node in cases of invalid code like this).

https://github.com/llvm/llvm-project/pull/69849