[all-commits] [llvm/llvm-project] 00cd6c: [Preprocessor] Reduce the memory overhead of `#def...

[Alex Lorenz via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 00cd6c04202acf71f74c670b2dd4343929d1f45f
      https://github.com/llvm/llvm-project/commit/00cd6c04202acf71f74c670b2dd4343929d1f45f
  Author: Alex Lorenz <arphaman at gmail.com>
  Date:   2022-02-14 (Mon, 14 Feb 2022)

  Changed paths:
    M clang/include/clang/Lex/MacroInfo.h
    M clang/lib/Lex/MacroInfo.cpp
    M clang/lib/Lex/PPDirectives.cpp
    M clang/lib/Serialization/ASTReader.cpp
    M clang/lib/Serialization/ASTWriter.cpp
    M clang/unittests/Lex/CMakeLists.txt
    A clang/unittests/Lex/PPMemoryAllocationsTest.cpp
    M lldb/source/Plugins/ExpressionParser/Clang/ClangModulesDeclVendor.cpp

  Log Message:
  -----------
  [Preprocessor] Reduce the memory overhead of `#define` directives (Recommit)

Recently we observed high memory pressure caused by clang during some parallel builds.
We discovered that we have several projects that have a large number of #define directives
in their TUs (on the order of millions), which caused huge memory consumption in clang due
to a lot of allocations for MacroInfo. We would like to reduce the memory overhead of
clang for a single #define to reduce the memory overhead for these files, to allow us to
reduce the memory pressure on the system during highly parallel builds. This change achieves
that by removing the SmallVector in MacroInfo and instead storing the tokens in an array
allocated using the bump pointer allocator, after all tokens are lexed.

The added unit test with 1000000 #define directives illustrates the problem. Prior to this
change, on arm64 macOS, clang's PP bump pointer allocator allocated 272007616 bytes, and
used roughly 272 bytes per #define. After this change, clang's PP bump pointer allocator
allocates 120002016 bytes, and uses only roughly 120 bytes per #define.

For an example test file that we have internally with 7.8 million #define directives, this
change produces the following improvement on arm64 macOS: Persistent allocation footprint for
this test case file as it's being compiled to LLVM IR went down 22% from 5.28 GB to 4.07 GB
and the total allocations went down 14% from 8.26 GB to 7.05 GB. Furthermore, this change
reduced the total number of allocations made by the system for this clang invocation from
1454853 to 133663, an order of magnitude improvement.

The recommit fixes the LLDB build failure.

Differential Revision: https://reviews.llvm.org/D117348