[PATCH] D124687: [Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to record and use pre-lexed preprocessor directive tokens, instead of minimized sources

[Argyrios Kyrtzidis via Phabricator via cfe-commits]

akyrtzi created this revision.
Herald added a subscriber: mgorny.
Herald added a project: All.
akyrtzi requested review of this revision.
Herald added a project: clang.
Herald added a subscriber: cfe-commits.

Make the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`, instead of implementation detail of `DependencyScanningFilesystem`. This has the following benefits:

- Full access to the preprocessor state during dependency scanning. E.g. a component can see what includes were taken and where they were located in the actual sources.
- Improved performance for dependency scanning. Measurements for dependency scanning of clang sources shows reduction in wall time
  - M1Pro: reduction of about -10.6%
  - iMacPro: reduction of about -14%
- Opportunity to use dependency scanning lexing to speed-up skipping of excluded conditional blocks during normal preprocessing (as follow-up, not part of this patch).

Since, after this change, we don't minimize sources and pass them in place of the real sources, `DependencyScanningFilesystem` is not technically necessary, but it has valuable performance benefits for caching file `stat`s along with the results of scanning the sources. So the setup of using the `DependencyScanningFilesystem` during a dependency scan remains.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D124687

Files:
  clang/include/clang/Basic/DiagnosticLexKinds.td
  clang/include/clang/Lex/DependencyDirectivesScanner.h
  clang/include/clang/Lex/DependencyDirectivesSourceMinimizer.h
  clang/include/clang/Lex/Lexer.h
  clang/include/clang/Lex/Preprocessor.h
  clang/include/clang/Lex/PreprocessorExcludedConditionalDirectiveSkipMapping.h
  clang/include/clang/Lex/PreprocessorOptions.h
  clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
  clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
  clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
  clang/lib/Frontend/FrontendActions.cpp
  clang/lib/Lex/CMakeLists.txt
  clang/lib/Lex/DependencyDirectivesScanner.cpp
  clang/lib/Lex/DependencyDirectivesSourceMinimizer.cpp
  clang/lib/Lex/Lexer.cpp
  clang/lib/Lex/PPDirectives.cpp
  clang/lib/Lex/PPLexerChange.cpp
  clang/lib/Lex/Preprocessor.cpp
  clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
  clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
  clang/test/ClangScanDeps/has_include_if_elif.cpp
  clang/test/ClangScanDeps/macro-expansions.cpp
  clang/test/ClangScanDeps/modulemap-via-vfs.m
  clang/test/ClangScanDeps/modules-fmodule-name-no-module-built.m
  clang/test/ClangScanDeps/modules-full-by-mod-name.cpp
  clang/test/ClangScanDeps/modules-full.cpp
  clang/test/ClangScanDeps/modules-inferred-explicit-build.m
  clang/test/ClangScanDeps/modules-inferred.m
  clang/test/ClangScanDeps/modules.cpp
  clang/test/ClangScanDeps/preprocess_minimized_pragmas.cpp
  clang/test/ClangScanDeps/regular_cdb.cpp
  clang/test/Lexer/minimize_source_to_dependency_directives_invalid_macro_name.c
  clang/test/Lexer/minimize_source_to_dependency_directives_pragmas.c
  clang/tools/clang-scan-deps/ClangScanDeps.cpp
  clang/unittests/Lex/DependencyDirectivesSourceMinimizerTest.cpp
  clang/unittests/Tooling/DependencyScannerTest.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D124687.426113.patch
Type: text/x-patch
Size: 157084 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20220429/902f243b/attachment-0001.bin>