[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov added inline comments.


================
Comment at: clang/lib/Lex/PPDirectives.cpp:1521
   Tok[0].setAnnotationValue(AnnotationVal);
-  EnterTokenStream(std::move(Tok), 1, true);
+  EnterTokenStream(std::move(Tok), 1, true, /*IsReinject*/ true);
 }
----------------
rsmith wrote:
> I think it'd be more useful to treat this as a new token. But that's not a strong preference.
Ah, there were too many changes and I missed this one. Definitely agree, clients can filter out annotation tokens themselves, if needed.


================
Comment at: clang/lib/Lex/Pragma.cpp:370
   // Push the tokens onto the stack.
-  EnterTokenStream(TokArray, PragmaToks.size(), true, true);
+  EnterTokenStream(TokArray, PragmaToks.size(), true, true,
+                   /*IsReinject*/ false);
----------------
rsmith wrote:
> I think this case is a reinjection; we've copied some tokens inside __pragma out into a duplicate position in the token stream. But I guess it doesn't matter because the tokens never escape the outer Lex function anyway.
Yeah, my logic is that it's not a re-injection in the sense that they were never the phase 4 tokens before.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:1136
     EnterTokenStream(std::move(ToksCopy), Toks.size(),
-                     /*DisableMacroExpansion*/ true);
+                     /*DisableMacroExpansion*/ true, IsReinject);
   };
----------------
rsmith wrote:
> I think this should always be `false`: the tokens we're producing here have never been emitted at `LexLevel` 0 before.
Ah, totally, we lexed the original tokens in this function...
Fixed, thanks!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885