[PATCH] D125487: [Tooling/DependencyScanning] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

[Argyrios Kyrtzidis via Phabricator via cfe-commits]

akyrtzi marked an inline comment as done.
akyrtzi added inline comments.


================
Comment at: clang/lib/Lex/DependencyDirectivesScanner.cpp:153
+
+  SmallVector<dependency_directives_scan::Token, 32> CurDirToks;
+  SmallVector<DirectiveWithTokens, 64> DirsWithToks;
----------------
jansvoboda11 wrote:
> Can you add a comment explaining the relationship between the members?
Done!


================
Comment at: clang/unittests/Lex/DependencyDirectivesScannerTest.cpp:114
+  EXPECT_EQ(cxx_export_module_decl, Directives[19].Kind);
+  EXPECT_EQ(cxx_import_decl, Directives[20].Kind);
+  EXPECT_EQ(pp_eof, Directives[21].Kind);
----------------
jansvoboda11 wrote:
> What's the reason being these changes?
Originally the directive 'tokens' were:
```
   cxx_export_decl,
   cxx_module_decl,
   cxx_import_decl,
```

While all the other token kinds were representing one top-level directive, `cxx_export_decl` was different in that it was treated like a "modifier" for `cxx_module_decl` or `cxx_import_decl`. But this inconsistency did not affect anything since the preprocessor was reading the minimized sources and the `cxx_*` tokens were generally ignored.

After these changes the preprocessor reads sets of ("one top-level directive kind" + "array of tokens for the directive") and I thought it's a bit better to keep all the directive kinds consistent as representing "one top-level directive", thus I made the change for directive kinds to be:
```
  cxx_module_decl,
  cxx_import_decl,
  cxx_export_module_decl,
  cxx_export_import_decl,
```
in which case "export module" is treated as a separate top-level directive kind than "module".


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D125487/new/

https://reviews.llvm.org/D125487