[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov marked an inline comment as not done.
ilya-biryukov added inline comments.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:956-957
   --LexLevel;
+  if (OnToken)
+    OnToken(Result, Source);
 }
----------------
ilya-biryukov wrote:
> ilya-biryukov wrote:
> > rsmith wrote:
> > > This seems like it's going to be extremely hard to use. If you just want the expanded token stream, then removing all the other calls to `OnToken` and only calling it here when `LexLevel == 0` will give you that.
> > I've tried `LexLevel == 0 && IsNewToken` and it **almost** works.
> > The only trouble left is dealing with the delayed parsing. E.g. I'm getting the callbacks for the same tokens multiple times in cases like parsing of method bodies.
> > 
> > Any ideas on how to best tackle this?
> > 
> > The example is:
> > ```
> > struct X {
> >   int method() { return 10; }
> >   int a = 10;
> > };
> > ```
> > 
> > Seeing multiple callbacks for `{ return 10; }` and `= 10;`, want to see only the first one.
> It almost works now, see the update revision.
> Almost no tokens get reported by the callback twice now, except the following case.
> Input:
> ```
> struct X {
>   int method() { return method(); }
> };
> ```
> After the body of the class, we get a an extra callback for `)` token from one of the functions under `ParsedLexedMethodDef`.
> This seems to be happening only for parenthesis.
More precisely, this happens from a following sequence of calls:

```
Lex() {LexLevel=0}
CachingLex() {LexLevel=1} // which does not have a token in `CachedTokens`, so it recurses into ...
Lex() {LexLevel=1}        // which results in a call to ...
CurTokenLexer->Lex()
```

At this point `CurTokenLexer` returns a token of a pre-lexed method definition, but the current implementation forgets that this was the case by the time the token is processed at the the top of the callstack (`Lex() {LexLevel=0}`).
So the token ends up being reported by a callback.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885