[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov marked 2 inline comments as done.
ilya-biryukov added a comment.

I'll try to explore bringing the overhead down.
The fact that `CachingLex` is happening at `LexLevel==1` might help, thanks for pointing that out!



================
Comment at: clang/lib/Lex/PPCaching.cpp:64
   ExitCachingLexMode();
-  Lex(Result);
+  Lex(Result, Report);
 
----------------
rsmith wrote:
> This change seems redundant: `LexLevel` is always `1` here, so this token would never be reported anyway. And with that gone I think you can remove the `Report` parameter entirely.
It is reported by the caller of `CachingLex` (the one that has `LexLevel == 0`) and it needs to know where the reported token comes from.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:892
   do {
     switch (CurLexerKind) {
     case CLK_Lexer:
----------------
rsmith wrote:
> Doesn't this always set `Report` to the same value as `IsNewToken`? (The only case we set `Report` to `false` is when we call `CachingLex` and it sets `IsNewToken` to `false`, and `CachingLex` can't be recursively called twice, so its recursive call to `Lex` can't set `Report` to `false`..)
We can have `IsNewToken = true && Report = false` when the recursive call to `Lex()` inside `CachingLex()` sets `Report = CurTokenLexer->isMacroExpansion()`.
This happens when we call `CachingLex` and the resulting token comes from a token stream for out-of-order parsing.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885