[all-commits] [llvm/llvm-project] b58a42: [Tooling/DependencyScanning] Rename refactorings t...

[Argyrios Kyrtzidis via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: b58a420ff4f92b085fd718600fda162059171a58
      https://github.com/llvm/llvm-project/commit/b58a420ff4f92b085fd718600fda162059171a58
  Author: Argyrios Kyrtzidis <kyrtzidis at apple.com>
  Date:   2022-05-26 (Thu, 26 May 2022)

  Changed paths:
    M clang/include/clang/Basic/DiagnosticLexKinds.td
    A clang/include/clang/Lex/DependencyDirectivesScanner.h
    R clang/include/clang/Lex/DependencyDirectivesSourceMinimizer.h
    M clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
    M clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
    M clang/lib/Frontend/FrontendActions.cpp
    M clang/lib/Lex/CMakeLists.txt
    A clang/lib/Lex/DependencyDirectivesScanner.cpp
    R clang/lib/Lex/DependencyDirectivesSourceMinimizer.cpp
    M clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
    M clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
    M clang/test/ClangScanDeps/has_include_if_elif.cpp
    A clang/test/ClangScanDeps/macro-expansions.cpp
    M clang/test/ClangScanDeps/modulemap-via-vfs.m
    M clang/test/ClangScanDeps/modules-fmodule-name-no-module-built.m
    M clang/test/ClangScanDeps/modules-full-by-mod-name.cpp
    M clang/test/ClangScanDeps/modules-full.cpp
    M clang/test/ClangScanDeps/modules-inferred-explicit-build.m
    M clang/test/ClangScanDeps/modules-inferred.m
    M clang/test/ClangScanDeps/modules.cpp
    M clang/test/ClangScanDeps/preprocess_minimized_pragmas.cpp
    M clang/test/ClangScanDeps/regular_cdb.cpp
    M clang/tools/clang-scan-deps/ClangScanDeps.cpp
    M clang/unittests/Lex/CMakeLists.txt
    A clang/unittests/Lex/DependencyDirectivesScannerTest.cpp
    R clang/unittests/Lex/DependencyDirectivesSourceMinimizerTest.cpp
    M clang/unittests/Tooling/DependencyScannerTest.cpp

  Log Message:
  -----------
  [Tooling/DependencyScanning] Rename refactorings towards transitioning dependency scanning to use pre-lexed preprocessor directive tokens

This is first of a series of patches for making the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`.
This patch only includes NFC renaming changes to make reviewing of the functionality changing parts easier.

Differential Revision: https://reviews.llvm.org/D125484


  Commit: b4c83a13f664582015ea22924b9a0c6290d41f5b
      https://github.com/llvm/llvm-project/commit/b4c83a13f664582015ea22924b9a0c6290d41f5b
  Author: Argyrios Kyrtzidis <kyrtzidis at apple.com>
  Date:   2022-05-26 (Thu, 26 May 2022)

  Changed paths:
    M clang/include/clang/Lex/DependencyDirectivesScanner.h
    M clang/include/clang/Lex/Lexer.h
    M clang/include/clang/Lex/Preprocessor.h
    R clang/include/clang/Lex/PreprocessorExcludedConditionalDirectiveSkipMapping.h
    M clang/include/clang/Lex/PreprocessorOptions.h
    M clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
    M clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
    M clang/lib/Frontend/FrontendActions.cpp
    M clang/lib/Lex/DependencyDirectivesScanner.cpp
    M clang/lib/Lex/Lexer.cpp
    M clang/lib/Lex/PPDirectives.cpp
    M clang/lib/Lex/PPLexerChange.cpp
    M clang/lib/Lex/Preprocessor.cpp
    M clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
    M clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
    M clang/test/Lexer/minimize_source_to_dependency_directives_invalid_macro_name.c
    M clang/test/Lexer/minimize_source_to_dependency_directives_pragmas.c
    M clang/unittests/Lex/DependencyDirectivesScannerTest.cpp
    M clang/unittests/Tooling/DependencyScannerTest.cpp

  Log Message:
  -----------
  [Tooling/DependencyScanning & Preprocessor] Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

This is a commit with the following changes:

* Remove `ExcludedPreprocessorDirectiveSkipMapping` and related functionality

Removes `ExcludedPreprocessorDirectiveSkipMapping`; its intended benefit for fast skipping of excluded directived blocks
will be superseded by a follow-up patch in the series that will use dependency scanning lexing for the same purpose.

* Refactor dependency scanning to produce pre-lexed preprocessor directive tokens, instead of minimized sources

Replaces the "source minimization" mechanism with a mechanism that produces lexed dependency directives tokens.

* Make the special lexing for dependency scanning a first-class feature of the `Preprocessor` and `Lexer`

This is bringing the following benefits:

    * Full access to the preprocessor state during dependency scanning. E.g. a component can see what includes were taken and where they were located in the actual sources.
    * Improved performance for dependency scanning. Measurements with a release+thin-LTO build shows ~ -11% reduction in wall time.
    * Opportunity to use dependency scanning lexing to speed-up skipping of excluded conditional blocks during normal preprocessing (as follow-up, not part of this patch).

For normal preprocessing measurements show differences are below the noise level.

Since, after this change, we don't minimize sources and pass them in place of the real sources, `DependencyScanningFilesystem` is not technically necessary, but it has valuable performance benefits for caching file `stat`s along with the results of scanning the sources. So the setup of using the `DependencyScanningFilesystem` during a dependency scan remains.

Differential Revision: https://reviews.llvm.org/D125486
Differential Revision: https://reviews.llvm.org/D125487
Differential Revision: https://reviews.llvm.org/D125488


Compare: https://github.com/llvm/llvm-project/compare/628b2bfad828...b4c83a13f664