[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

Thu May 9 08:48:06 PDT 2019

ilya-biryukov added a comment.

Here are some numbers from running `clang -cc1 -Eonly` on `SemaExpr.cpp`, it includes a whole ton of headers, including a rather large `TreeTransform.h`. This was run on my machine, so keep in mind it's rather noisy. Minimal times should be somewhat representative, though.

  Before:
    Time (mean ± σ):     159.5 ms ±   7.3 ms    [User: 137.3 ms, System: 22.1 ms]
    Range (min … max):   148.9 ms … 199.5 ms    100 runs

  After (no callback specified):
    Time (mean ± σ):     165.0 ms ±   7.5 ms    [User: 142.8 ms, System: 22.2 ms]
    Range (min … max):   155.0 ms … 196.7 ms    100 runs

And two more measurements with callbacks non-empty callbacks:

  Counting tokens:
    Time (mean ± σ):     169.0 ms ±   9.5 ms    [User: 147.3 ms, System: 21.7 ms]
    Range (min … max):   158.3 ms … 206.0 ms    100 runs

  Saving tokens to a vector:
    Time (mean ± σ):     191.0 ms ±   5.5 ms    [User: 154.1 ms, System: 36.8 ms]
    Range (min … max):   180.4 ms … 221.0 ms    100 runs

Overall, the performance cost in empty-callback case seems to be lower than `5%`. This is significant, but hopefully acceptable.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885