[all-commits] [llvm/llvm-project] 775700: [clang-doc] Merge data into persistent memory

[Paul Kirth via All-commits]

  Branch: refs/heads/users/ilovepi/new-arenas-13
  Home:   https://github.com/llvm/llvm-project
  Commit: 7757006508bcde8834cc1ab1cfe2ebaed468d090
      https://github.com/llvm/llvm-project/commit/7757006508bcde8834cc1ab1cfe2ebaed468d090
  Author: Paul Kirth <paulkirth at google.com>
  Date:   2026-04-01 (Wed, 01 Apr 2026)

  Changed paths:
    M clang-tools-extra/clang-doc/Representation.cpp
    M clang-tools-extra/clang-doc/Representation.h
    M clang-tools-extra/clang-doc/tool/ClangDocMain.cpp

  Log Message:
  -----------
  [clang-doc] Merge data into persistent memory

We have a need for persistent memory for the final info. Since each
group processes a single USR at a time, every USR is only ever processed by
a single thread from the thread pool. This means that we can keep per
thread persistent storage for all the info. There is significant
duplicated data between all the serialized records, so we can just merge
the final/unique items into the persistent arena, and clear out the
scratch/transient arena as we process each record in the bitcode.

The patch adds some APIs to help with managing the data, merging, and
allocation of data in the correct arena. It also safely merges and deep
copies data from the transient arenas into persistent storage that is
never reset until the program completes.

This patch reduces memory by another % over the previous patches,
bringing the total savings over the baseline to 57%. Runtime performance
and benchmarks stay mostly flat with modest improvements.

| Metric | Baseline | Prev | This | Culm% | Seq% |
| :--- | :--- | :--- | :--- | :--- | :--- |
| Time | 920.5s | 991.5s | 987.2s | +7.2% | -0.4% |
| Memory | 86.0G | 40.0G | 36.9G | -57.1% | -8.0% |

| Benchmark | Baseline | Prev | This | Culm% | Seq% |
| :--- | :--- | :--- | :--- | :--- | :--- |
| BM_BitcodeReader_Scale/10 | 67.9us | 72.2us | 72.2us | +6.3% | -0.0% |
| BM_BitcodeReader_Scale/10000 | 70.5ms | 22.5ms | 17.3ms | -75.5% | -23.2% |
| BM_BitcodeReader_Scale/4096 | 23.2ms | 6.6ms | 7.1ms | -69.5% | +7.4% |
| BM_BitcodeReader_Scale/512 | 509.4us | 898.7us | 550.5us | +8.1% | -38.7% |
| BM_BitcodeReader_Scale/64 | 114.8us | 133.7us | 120.8us | +5.2% | -9.6% |
| BM_EmitInfoFunction | 1.6us | 1.9us | 1.8us | +12.9% | -3.3% |
| BM_Index_Insertion/10 | 2.3us | 4.1us | 3.5us | +52.4% | -14.7% |
| BM_Index_Insertion/10000 | 3.1ms | 5.3ms | 4.8ms | +53.5% | -10.0% |
| BM_Index_Insertion/4096 | 1.3ms | 2.1ms | 1.9ms | +50.8% | -9.2% |
| BM_Index_Insertion/512 | 153.6us | 251.8us | 227.0us | +47.8% | -9.9% |
| BM_Index_Insertion/64 | 18.1us | 30.2us | 26.7us | +47.9% | -11.7% |
| BM_JSONGenerator_Scale/10000 | 89.6ms | 81.4ms | 83.4ms | -6.9% | +2.5% |
| BM_JSONGenerator_Scale/4096 | 33.7ms | 31.0ms | 32.4ms | -3.9% | +4.5% |
| BM_Mapper_Scale/10000 | 104.3ms | 112.3ms | 103.5ms | -0.8% | -7.9% |
| BM_Mapper_Scale/4096 | 44.3ms | 45.0ms | 43.8ms | -1.2% | -2.5% |
| BM_Mapper_Scale/512 | 7.6ms | 7.7ms | 7.5ms | -1.3% | -2.4% |
| BM_Mapper_Scale/64 | 3.1ms | 3.0ms | 3.0ms | -1.9% | -0.3% |
| BM_MergeInfos_Scale/10000 | 12.2ms | 575.6us | 500.1us | -95.9% | -13.1% |
| BM_MergeInfos_Scale/2 | 1.9us | 1.8us | 1.8us | -4.4% | -1.7% |
| BM_MergeInfos_Scale/4096 | 2.8ms | 205.3us | 200.4us | -92.8% | -2.4% |
| BM_MergeInfos_Scale/512 | 68.9us | 20.5us | 19.5us | -71.7% | -5.1% |
| BM_MergeInfos_Scale/64 | 10.3us | 3.8us | 4.0us | -60.9% | +4.8% |
| BM_MergeInfos_Scale/8 | 2.8us | 1.9us | 1.9us | -31.7% | -1.8% |
| BM_SerializeFunctionInfo | 25.5us | 25.8us | 26.1us | +2.2% | +1.3% |



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications