[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Richard Smith - zygoloid via Phabricator via cfe-commits]

rsmith added a comment.

I'd like to understand more about the intended use cases of this functionality. What information do clients want?



================
Comment at: clang/lib/Lex/Preprocessor.cpp:870-900
+  TokenSource Source;
   do {
+    Source = TokenSource();
+
     switch (CurLexerKind) {
     case CLK_Lexer:
+      Source.InDirective = CurLexer->ParsingPreprocessorDirective;
----------------
This is a lot of extra stuff to be doing in the main `Lex` loop. Adding one (perfectly-predicted) branch on `OnToken` seems like it should be fine, but this seems like a bit more added complexity than I'd prefer. I'd like some performance measurements of `-cc1 -Eonly` to see if this makes any real difference.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:896
     case CLK_LexAfterModuleImport:
-      ReturnedToken = LexAfterModuleImport(Result);
+      Source.InDirective = true;
+
----------------
This isn't a directive; these are normal tokens.


================
Comment at: clang/lib/Lex/Preprocessor.cpp:956-957
   --LexLevel;
+  if (OnToken)
+    OnToken(Result, Source);
 }
----------------
This seems like it's going to be extremely hard to use. If you just want the expanded token stream, then removing all the other calls to `OnToken` and only calling it here when `LexLevel == 0` will give you that.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885