[PATCH] D67127: [clang-scan-deps] add skip excluded conditional preprocessor block preprocessing optimization

[Alex Lorenz via Phabricator via cfe-commits]

arphaman created this revision.
arphaman added reviewers: Bigcheese, dexonsmith, aganea.
Herald added subscribers: ributzka, tschuett, jsji, jkorous, MaskRay, kbarton, mgorny, nemanjai.
Herald added a project: clang.

This patch adds an optimization to clang-scan-deps and clang's preprocessor that skips excluded preprocessor blocks by bumping the lexer pointer, and not lexing the tokens until reaching appropriate `#else/#endif` directive. The skip positions and lexer offsets are computed when the file is minimized, directly from the minimized tokens.

On an 18-core iMacPro with macOS Catalina Beta I got 10-15% speed-up from this optimization when running `clang-scan-deps` on the compilation database for a recent LLVM and Clang (3511 files).


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D67127

Files:
  clang/include/clang/Frontend/CompilerInstance.h
  clang/include/clang/Lex/DependencyDirectivesSourceMinimizer.h
  clang/include/clang/Lex/Lexer.h
  clang/include/clang/Lex/PPCallbacks.h
  clang/include/clang/Tooling/DependencyScanning/DependencyScanningFilesystem.h
  clang/include/clang/Tooling/DependencyScanning/DependencyScanningService.h
  clang/include/clang/Tooling/DependencyScanning/DependencyScanningWorker.h
  clang/include/clang/Tooling/DependencyScanning/PPRangeSkipping.h
  clang/lib/Frontend/CompilerInstance.cpp
  clang/lib/Lex/DependencyDirectivesSourceMinimizer.cpp
  clang/lib/Lex/Lexer.cpp
  clang/lib/Lex/PPDirectives.cpp
  clang/lib/Tooling/DependencyScanning/CMakeLists.txt
  clang/lib/Tooling/DependencyScanning/DependencyScanningFilesystem.cpp
  clang/lib/Tooling/DependencyScanning/DependencyScanningService.cpp
  clang/lib/Tooling/DependencyScanning/DependencyScanningWorker.cpp
  clang/lib/Tooling/DependencyScanning/PPRangeSkipping.cpp
  clang/test/ClangScanDeps/regular_cdb.cpp
  clang/tools/clang-scan-deps/ClangScanDeps.cpp
  clang/unittests/Lex/DependencyDirectivesSourceMinimizerTest.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D67127.218531.patch
Type: text/x-patch
Size: 29899 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/cfe-commits/attachments/20190903/89f4c5dd/attachment-0001.bin>