[all-commits] [llvm/llvm-project] 6a1f50: [clang][deps] Prune unused header search paths

[Jan Svoboda via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 6a1f50b84ae8f8a8087fcdbe5f27dae8c76878f1
      https://github.com/llvm/llvm-project/commit/6a1f50b84ae8f8a8087fcdbe5f27dae8c76878f1
  Author: Jan Svoboda <jan_svoboda at apple.com>
  Date:   2021-10-12 (Tue, 12 Oct 2021)

  Changed paths:
    M clang/include/clang/Serialization/ASTBitCodes.h
    M clang/include/clang/Serialization/ModuleFile.h
    M clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
    M clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
    M clang/include/clang/Tooling/DependencyScanning/ModuleDepCollector.h
    M clang/lib/Serialization/ASTReader.cpp
    M clang/lib/Serialization/ASTWriter.cpp
    M clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
    M clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
    M clang/lib/Tooling/DependencyScanning/ModuleDepCollector.cpp
    A clang/test/ClangScanDeps/Inputs/header-search-pruning/a/a.h
    A clang/test/ClangScanDeps/Inputs/header-search-pruning/b/b.h
    A clang/test/ClangScanDeps/Inputs/header-search-pruning/begin/begin.h
    A clang/test/ClangScanDeps/Inputs/header-search-pruning/cdb.json
    A clang/test/ClangScanDeps/Inputs/header-search-pruning/end/end.h
    A clang/test/ClangScanDeps/Inputs/header-search-pruning/mod.h
    A clang/test/ClangScanDeps/Inputs/header-search-pruning/module.modulemap
    A clang/test/ClangScanDeps/header-search-pruning.cpp
    M clang/tools/clang-scan-deps/ClangScanDeps.cpp

  Log Message:
  -----------
  [clang][deps] Prune unused header search paths

To reduce the number of explicit builds of a single module, we can try to squash multiple occurrences of the module with different command-lines (and context hashes) by removing benign command-line options. The greatest contributors to benign differences between command-lines are the header search paths.

In this patch, the lookup cache in `HeaderSearch` is used to identify paths that were actually used when implicitly building the module during scanning. This information is serialized into the unhashed control block of the implicitly-built PCM. The dependency scanner then loads this and may use it to prune the header search paths before computing the context hash of the module and generating the command-line.

We could also prune the header search paths when serializing `HeaderSearchOptions` into the PCM. That way, we could do it only once instead of every load of the PCM file by dependency scanner. However, that would result in a PCM file whose contents don't produce the same context hash as the original build, which is probably highly surprising.

There is an alternative approach to storing extra information into the PCM: wire up preprocessor callbacks to capture the used header search paths on-the-fly during preprocessing of modularized headers (similar to what we currently do for the main source file and textual headers). Right now, that's not compatible with the fact that we do an actual implicit build producing PCM files during dependency scanning. The second run of dependency scanner loads the PCM from the first run, skipping the preprocessing altogether, which would result in different results between runs. We can revisit this approach when we stop building implicitly during dependency scanning.

Depends on D102923.

Reviewed By: dexonsmith

Differential Revision: https://reviews.llvm.org/D102488