[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Richard Smith - zygoloid via Phabricator via cfe-commits]

rsmith added a comment.

Thinking about this some more: distinguishing between "macro expansion" and other cases seems like a proxy for "this token came from inside the preprocessor / lexer" versus "this token was provided by the user", which is also exactly what `IsNewToken` is supposed to capture. I don't think we need two separate flags here.

I also suspect that a significant amount of the cost is the additional function call on the hot path (the 1-parameter `Lex` wrapper function). I wonder if there's something that we could do to remove that from the hot path without duplicating the whole `Lex` function. Maybe we could add a flag to `Token`s to say that they're a replay of prior tokens, rather than being newly-created tokens, and make `TokenLexer` set that flag on tokens it produces if its token sequence came from a "replay" context (that's largely the same as your current "not a macro expansion" check, but there are probably cases where the preprocessor itself generates tokens for which that's wrong; we should have a flag on `EnterTokenStream` to indicate which mode we're in). We can then also change `CachingLex` to set the flag, and get rid of the `IsNewToken` flag too.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885