[PATCH] D20131: Fixed crash during code completion in file included within declaration

[Cameron via cfe-commits]

cameron314 added a comment.

I fixed this last July so the details are a little hazy, but fortunately it turns out I documented what I had found:

> It seems when clang parses an enum declaration, it first parses the declaration specifiers, then stops if a semicolon is present.

>  The trouble is, an #include could cause the file to change part-way through the declaration. If completion is enabled, then when leaving the included file in which the completion was requested, the tokenizer is cleared (clang tries to simulate an EOF at that point to prevent further parsing).

>  But, while the tokenizer is cleared, the rest of the declaration still needs to be parsed, and the peeked token (a semicolon here) can no longer be advanced over, since it refers in this case to a token in a tokenizer that has been freed. Boom, null reference exception!

>  It seems changing the lexer kind to the caching lexer when the artificial EOF is inserted does the right thing in this case – a cached EOF token is returned on the advancement of the peeked semicolon.


I tried to add a test; I can reproduce the crash, and with this patch it no longer crashes, but because there's a subsequent parsing error after the completion point, when passing via `clang.exe` only the error diagnostics are printed without any of the completions (compared to libclang which allows you to retrieve both the completions and the diagnostics). This means I cannot write a RUN command in the lit test without it failing (the exit code is non-zero). I have no way to distinguish between a crash and the normal errors (with hidden completions). Is there any other way to test this?


================
Comment at: lib/Lex/PPLexerChange.cpp:380-382
@@ -379,3 +379,5 @@
         CurLexer->FormTokenWithChars(Result, CurLexer->BufferEnd, tok::eof);
         CurLexer.reset();
+        if (CurLexerKind == CLK_Lexer)
+          CurLexerKind = CLK_CachingLexer;
       } else {
----------------
rsmith wrote:
> Can you use `CurLexer->cutOffLexing()` instead?
Nope, still crashes when I try (before the reset, obviously).


http://reviews.llvm.org/D20131