[all-commits] [llvm/llvm-project] 8d9db9: Reland "[ThinLTO] Populate declaration import stat...

[Mingming Liu via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 8d9db947b725fefbb02905c5d6be05e09a306f6b
      https://github.com/llvm/llvm-project/commit/8d9db947b725fefbb02905c5d6be05e09a306f6b
  Author: Mingming Liu <mingmingl at google.com>
  Date:   2024-06-20 (Thu, 20 Jun 2024)

  Changed paths:
    M llvm/include/llvm/IR/ModuleSummaryIndex.h
    M llvm/include/llvm/Transforms/IPO/FunctionImport.h
    M llvm/lib/LTO/LTO.cpp
    M llvm/lib/LTO/LTOBackend.cpp
    M llvm/lib/Transforms/IPO/FunctionImport.cpp
    M llvm/test/ThinLTO/X86/funcimport-stats.ll
    A llvm/test/ThinLTO/X86/import_callee_declaration.ll
    M llvm/test/Transforms/FunctionImport/funcimport.ll
    M llvm/tools/llvm-link/llvm-link.cpp

  Log Message:
  -----------
  Reland "[ThinLTO] Populate declaration import status except for distributed ThinLTO under a default-off new option" (#95482)

Make `FunctionsToImportTy` an `unordered_map` rather than `DenseMap`.
Credit goes to jvoung@ for the 'DenseMap -> unordered_map' change. This
is a reland of https://github.com/llvm/llvm-project/pull/92718

* `DenseMap` allocates space for a large number of key/value pairs and
wastes space when the number of elements are small.
* While init bucket size is zero [1], it quickly allocates buckets for 64 elements [2]
when the number of elements is small (for example, 3 or 4 elements). The programmer
manual [3] also mentions it could waste space.
* Experiments show `FunctionsToImportTy.size()` is smaller than 4 for
multiple binaries with high indexing ram usage. `unordered_map` grows
factor is at most 2 in llvm libc [4] for insert operations.
 
With this change, `ComputeCrossModuleImport` ram increase is smaller
than 0.5G on a couple of binaries with high indexing ram usage. A wider
range of (pre-release) tests pass.

[1] https://github.com/llvm/llvm-project/blob/ad79a14c9e5ec4a369eed4adf567c22cc029863f/llvm/include/llvm/ADT/DenseMap.h#L431-L432 
[2] https://github.com/llvm/llvm-project/blob/ad79a14c9e5ec4a369eed4adf567c22cc029863f/llvm/include/llvm/ADT/DenseMap.h#L849
[3] https://llvm.org/docs/ProgrammersManual.html#llvm-adt-densemap-h
[4] https://github.com/llvm/llvm-project/blob/ad79a14c9e5ec4a369eed4adf567c22cc029863f/libcxx/include/__hash_table#L1525-L1526

**Original commit message** 
The goal is to populate `declaration` import status if a new flag
`-import-declaration` is on.

* For in-process ThinLTO, the `declaration` status is visible to backend
`function-import` pass, so `FunctionImporter::importFunctions` should
read the import status and be no-op for declaration summaries.
Basically, the postlink pipeline is updated to keep its current behavior
(import definitions), but not updated to handle `declaration` summaries.
Two use cases ([better call-graph
sort](https://discourse.llvm.org/t/rfc-for-better-call-graph-sort-build-a-more-complete-call-graph-by-adding-more-indirect-call-edges/74029#support-cross-module-function-declaration-import-5)
or [cross-module
auto-init](https://github.com/llvm/llvm-project/pull/87597#discussion_r1556067195))
would use this bit differently.

* For distributed ThinLTO, the `declaration` status is not serialized to
bitcode. As discussed, https://github.com/llvm/llvm-project/pull/87600
will do this.



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications