[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov added a comment.

The suggested approach looks promising. The difference seems to be within the noise levels on my machine:

  Before the change (baseline upstream revision):
    Time (mean ± σ):     159.1 ms ±   8.7 ms    [User: 138.3 ms, System: 20.6 ms]
    Range (min … max):   150.4 ms … 196.2 ms    100 runs
  
  After (no callback specified):
    Time (mean ± σ):     160.4 ms ±   7.6 ms    [User: 138.5 ms, System: 21.7 ms]
    Range (min … max):   151.0 ms … 191.5 ms    100 runs

I'm sending out a prototype I used for measures for review. The flag is currently set by the preprocessor, but I guess it would make more sense to do the following before landing this:

- flip the flag, i.e. instead of reporting whether the token is "new" (the common case), report whether it's "cached", i.e. coming from `CachedTokens` or from a non-macro-expansion token stream.
- set this flag only inside `CachingLex` and `TokenLexer`.

The revision also removes `IsNewToken` from `CachingLex` and sets the flag in the token instead.
Any other suggestions?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885