[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov marked an inline comment as not done.
ilya-biryukov added inline comments.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:956-957
   --LexLevel;
+  if (OnToken)
+    OnToken(Result, Source);
 }
----------------
ilya-biryukov wrote:
> rsmith wrote:
> > This seems like it's going to be extremely hard to use. If you just want the expanded token stream, then removing all the other calls to `OnToken` and only calling it here when `LexLevel == 0` will give you that.
> I've tried `LexLevel == 0 && IsNewToken` and it **almost** works.
> The only trouble left is dealing with the delayed parsing. E.g. I'm getting the callbacks for the same tokens multiple times in cases like parsing of method bodies.
> 
> Any ideas on how to best tackle this?
> 
> The example is:
> ```
> struct X {
>   int method() { return 10; }
>   int a = 10;
> };
> ```
> 
> Seeing multiple callbacks for `{ return 10; }` and `= 10;`, want to see only the first one.
It almost works now, see the update revision.
Almost no tokens get reported by the callback twice now, except the following case.
Input:
```
struct X {
  int method() { return method(); }
};
```
After the body of the class, we get a an extra callback for `)` token from one of the functions under `ParsedLexedMethodDef`.
This seems to be happening only for parenthesis.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885