[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov marked an inline comment as done.
ilya-biryukov added inline comments.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:956-957
   --LexLevel;
+  if (OnToken)
+    OnToken(Result, Source);
 }
----------------
rsmith wrote:
> This seems like it's going to be extremely hard to use. If you just want the expanded token stream, then removing all the other calls to `OnToken` and only calling it here when `LexLevel == 0` will give you that.
I've tried `LexLevel == 0 && IsNewToken` and it **almost** works.
The only trouble left is dealing with the delayed parsing. E.g. I'm getting the callbacks for the same tokens multiple times in cases like parsing of method bodies.

Any ideas on how to best tackle this?

The example is:
```
struct X {
  int method() { return 10; }
  int a = 10;
};
```

Seeing multiple callbacks for `{ return 10; }` and `= 10;`, want to see only the first one.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885