[PATCH] D59885: [Lex] Allow to consume tokens while preprocessing

[Ilya Biryukov via Phabricator via cfe-commits]

ilya-biryukov marked an inline comment as done.
ilya-biryukov added a comment.

In D59885#1485159 <https://reviews.llvm.org/D59885#1485159>, @rsmith wrote:

> I'd like to understand more about the intended use cases of this functionality. What information do clients want?


We are aiming to collect all expanded tokens, all raw tokens and how they relate to each other (i.e. which raw tokens form macro calls, #include directives, etc that produced particular subrange of expanded tokens).
We go through all this trouble to allow mapping expanded tokens (which is what you "see" in the AST, even though AST does not store them) back to the raw tokens that produced them (which you can finally map to textual offsets, the final output in various tooling-related scenarios).
That information is used specifically to map (an allowed subset of) changes in the expanded token stream to textual changes.

We could actually get most of this information merely by looking at source locations. The only bit that we're missing is how to map from a macro call tokens into **all** expanded tokens that were produced by it.
I'll play around with reporting only the expanded token stream here and using source locations to produce the rest of the information we need.



================
Comment at: clang/lib/Lex/Preprocessor.cpp:870-900
+  TokenSource Source;
   do {
+    Source = TokenSource();
+
     switch (CurLexerKind) {
     case CLK_Lexer:
+      Source.InDirective = CurLexer->ParsingPreprocessorDirective;
----------------
rsmith wrote:
> This is a lot of extra stuff to be doing in the main `Lex` loop. Adding one (perfectly-predicted) branch on `OnToken` seems like it should be fine, but this seems like a bit more added complexity than I'd prefer. I'd like some performance measurements of `-cc1 -Eonly` to see if this makes any real difference.
Happy to do the measurements. Do you have a good collection of input files for this in mind?


================
Comment at: clang/lib/Lex/Preprocessor.cpp:896
     case CLK_LexAfterModuleImport:
-      ReturnedToken = LexAfterModuleImport(Result);
+      Source.InDirective = true;
+
----------------
rsmith wrote:
> This isn't a directive; these are normal tokens.
Sorry if it's a silly question, just wanted to clarify I'm not missing anything here.

These tokens are the name of the module,  "import-suffix" and the semicolon that follows it?
And they end up being used to build the AST for `ImportDecl`?


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D59885/new/

https://reviews.llvm.org/D59885