[lld] [LTO][ELF][lld] Use unique string saver in ELF bitcode symbol parsing (PR #106670)

Mingming Liu via llvm-commits llvm-commits at lists.llvm.org
Thu Sep 5 00:47:46 PDT 2024


================
@@ -56,6 +57,10 @@ bool hasContext();
 
 inline llvm::StringSaver &saver() { return context().saver; }
----------------
minglotus-6 wrote:

UniqueStringSaver allocates a `DenseMap<StringRef>` for de-duplication, and the dense-set has a memory overhead when the saved strings don't get duplicated.  Symbols get duplicated (due to `#include` and `linkonce_odr`) a lot to offset the dense-set overhead, while other saved strings (e.g., file paths) don't get much duplication. Presumably the number of duplicated symbols are large enough so that switching this one to unique saver is more memory efficient than HEAD but less memory efficient than using unique saver selectively for symbols (experiment data below)

Besides, the current saver are used in a couple of places (in lld, llvm-objdump, etc) and it'd be easier to use a couple of incremental changes to make every user of `saver` compile (to reap the memory saving for indexing first) even if there are more places where unique saver is better. I added a FIXME in CommonLinkerContext.h to ook into other places where duplications are common in saved strings and unique saver make sense.

##### Experiment Data
Experiment data are collected from two benchmarks x three set-ups, showing that selectively uniqufy symbols has a small win over switching.

* The peak indexing memory, measured before indexing returns https://github.com/llvm/llvm-project/blob/77f04882251b1e44239d6d7545cd62301e903a4a/lld/ELF/LTO.cpp#L356-L357 

  |benchmark                               |search1|search2|
  |----------------------------------------|-------|-------|
  |HEAD                                    |3.4GiB |9.89GiB|
  |switching to unique saver               |3.37GiB|8.79GiB| 
  |use unique saver for BitcodeFile methods |3.30GiB|8.48GiB|

* The peak memory under `lld::elf::parseFiles` (where bitcode parsing happens), collected from the same run. 
   |benchmark                               |search1|search2 |
  |----------------------------------------|-------|--------|
  |HEAD                                    |1.1GiB |2.79GiB |
  |switching to unique saver               |921MiB |2.01GiB |
  |use unique saver for BitcodeFile methods|911MiB |1.991GiB|
  




https://github.com/llvm/llvm-project/pull/106670


More information about the llvm-commits mailing list