[all-commits] [llvm/llvm-project] 76dd74: [CGData] Lazy loading support for stable function ...

[Zhaoxuan Jiang via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 76dd742f7b32e4d3acf50fab1dbbd897f215837e
      https://github.com/llvm/llvm-project/commit/76dd742f7b32e4d3acf50fab1dbbd897f215837e
  Author: Zhaoxuan Jiang <jiangzhaoxuan94 at gmail.com>
  Date:   2025-08-14 (Thu, 14 Aug 2025)

  Changed paths:
    M llvm/include/llvm/CGData/CodeGenData.h
    M llvm/include/llvm/CGData/CodeGenData.inc
    M llvm/include/llvm/CGData/StableFunctionMap.h
    M llvm/include/llvm/CGData/StableFunctionMapRecord.h
    M llvm/lib/CGData/CodeGenData.cpp
    M llvm/lib/CGData/CodeGenDataReader.cpp
    M llvm/lib/CGData/StableFunctionMap.cpp
    M llvm/lib/CGData/StableFunctionMapRecord.cpp
    M llvm/lib/CodeGen/GlobalMergeFunctions.cpp
    M llvm/test/ThinLTO/AArch64/cgdata-merge-write.ll
    M llvm/test/tools/llvm-cgdata/empty.test
    M llvm/test/tools/llvm-cgdata/error.test
    M llvm/test/tools/llvm-cgdata/merge-combined-funcmap-hashtree.test
    M llvm/test/tools/llvm-cgdata/merge-funcmap-archive.test
    M llvm/test/tools/llvm-cgdata/merge-funcmap-concat.test
    M llvm/test/tools/llvm-cgdata/merge-funcmap-double.test
    M llvm/test/tools/llvm-cgdata/merge-funcmap-single.test
    M llvm/tools/llvm-cgdata/Opts.td
    M llvm/tools/llvm-cgdata/llvm-cgdata.cpp
    M llvm/unittests/CGData/StableFunctionMapTest.cpp

  Log Message:
  -----------
  [CGData] Lazy loading support for stable function map (#151660)

The stable function map could be huge for a large application. Fully
loading it is slow and consumes a significant amount of memory, which is
unnecessary and drastically slows down compilation especially for
non-LTO and distributed-ThinLTO setups. This patch introduces an opt-in
lazy loading support for the stable function map. The detailed changes
are:

- `StableFunctionMap`
- The map now stores entries in an `EntryStorage` struct, which includes
offsets for serialized entries and a `std::once_flag` for thread-safe
lazy loading.
- The underlying map type is changed from `DenseMap` to
`std::unordered_map` for compatibility with `std::once_flag`.
- `contains()`, `size()` and `at()` are implemented to only load
requested entries on demand.

- Lazy Loading Mechanism
- When reading indexed codegen data, if the newly-introduced
`-indexed-codegen-data-lazy-loading` flag is set, the stable function
map is not fully deserialized up front. The binary format for the stable
function map now includes offsets and sizes to support lazy loading.
- The safety of lazy loading is guarded by the once flag per function
hash. This guarantees that even in a multi-threaded environment, the
deserialization for a given function hash will happen exactly once. The
first thread to request it performs the load, and subsequent threads
will wait for it to complete before using the data. For single-threaded
builds, the overhead is negligible (a single check on the once flag).
For multi-threaded scenarios, users can omit the flag to retain the
previous eager-loading behavior.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications