[llvm] [IR] Store metadata uses in a vector instead of a map (PR #227186)
Fangrui Song via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 28 20:50:34 PDT 2026
https://github.com/MaskRay created https://github.com/llvm/llvm-project/pull/227186
ReplaceableUses records uses in
```
SmallDenseMap<void *, std::pair<OwnerTy, uint64_t>, 4>
```
where the uint64_t orders uses for RAUW (21909e35cb06). A bucket is 24 bytes and DenseMap floors its first out-of-line table at 64 buckets, so a node with three uses allocates 1536 bytes. Debug info creates such nodes by the hundred thousand: while IRGen runs, every uniqued node that references a DIBuilder temporary has one.
Store the uses in a SmallVector instead. Dropping a use moves the last entry into its slot, and a list that grows past 32 entries gets a Ref-to-index DenseMap, so that dropping any use takes constant time. This matters because assignment tracking drains the dbg_assign uses of an alloca oldest first. A function with 50,000 assignments to one local compiles in 9.3 s before and after this change, and in 10.7 s with a vector that keeps insertion order, where every drop shifts the tail.
RAUW visits uses, and findDbgValues returns records, in the order of the list rather than the order they were added in. Sibling debug records at one point and the numbering of metadata can therefore come out differently. The object code below is unchanged.
Measured with clang on preprocessed SLPVectorizer.cpp / X86ISelLowering.cpp, object code byte-identical at -O2 -g and -O0 -g:
```
max RSS instructions:u
-O2 -g 1366 -> 1162 MiB (-14.9%) -0.31%
870 -> 824 MiB ( -5.2%) -0.32%
-O3 -g -flto=thin 1296 -> 1155 MiB (-10.9%) -0.37%
867 -> 803 MiB ( -7.3%) -0.35%
-O0 -g 1201 -> 1117 MiB ( -7.0%) -0.09%
805 -> 771 MiB ( -4.3%) -0.04%
```
The existing tests only exercise lists of up to three references. Add a test that drops, moves and re-adds references of an indexed list and looks each one up again.
LLM-aided
>From 134a8005a3385d50070ab64869ba64baa7eeb757 Mon Sep 17 00:00:00 2001
From: Fangrui Song <i at maskray.me>
Date: Sun, 20 Sep 2026 23:43:34 -0700
Subject: [PATCH] [IR] Store metadata uses in a vector instead of a map
ReplaceableUses records uses in
```
SmallDenseMap<void *, std::pair<OwnerTy, uint64_t>, 4>
```
where the uint64_t orders uses for RAUW (21909e35cb06). A bucket is 24
bytes and DenseMap floors its first out-of-line table at 64 buckets, so
a node with three uses allocates 1536 bytes. Debug info creates such
nodes by the hundred thousand: while IRGen runs, every uniqued node that
references a DIBuilder temporary has one.
Store the uses in a SmallVector instead. Dropping a use moves the last
entry into its slot, and a list that grows past 32 entries gets a
Ref-to-index DenseMap, so that dropping any use takes constant time.
This matters because assignment tracking drains the dbg_assign uses of
an alloca oldest first. A function with 50,000 assignments to one local
compiles in 9.3 s before and after this change, and in 10.7 s with a
vector that keeps insertion order, where every drop shifts the tail.
RAUW visits uses, and findDbgValues returns records, in the order of
the list rather than the order they were added in. Sibling debug records
at one point and the numbering of metadata can therefore come out
differently. The object code below is unchanged.
Measured with clang on preprocessed SLPVectorizer.cpp /
X86ISelLowering.cpp, object code byte-identical at -O2 -g and -O0 -g:
```
max RSS instructions:u
-O2 -g 1366 -> 1162 MiB (-14.9%) -0.31%
870 -> 824 MiB ( -5.2%) -0.32%
-O3 -g -flto=thin 1296 -> 1155 MiB (-10.9%) -0.37%
867 -> 803 MiB ( -7.3%) -0.35%
-O0 -g 1201 -> 1117 MiB ( -7.0%) -0.09%
805 -> 771 MiB ( -4.3%) -0.04%
```
The existing tests only exercise lists of up to three references. Add a
test that drops, moves and re-adds references of an indexed list and
looks each one up again.
LLM-aided
---
llvm/include/llvm/IR/Metadata.h | 24 ++++-
llvm/lib/IR/Metadata.cpp | 165 ++++++++++++++---------------
llvm/unittests/IR/MetadataTest.cpp | 23 ++++
3 files changed, 124 insertions(+), 88 deletions(-)
diff --git a/llvm/include/llvm/IR/Metadata.h b/llvm/include/llvm/IR/Metadata.h
index 9347a3dd93e13..e0f82951dcbe4 100644
--- a/llvm/include/llvm/IR/Metadata.h
+++ b/llvm/include/llvm/IR/Metadata.h
@@ -397,12 +397,27 @@ class ReplaceableUses {
using OwnerTy = MetadataTracking::OwnerTy;
private:
- uint64_t NextIndex = 0;
- SmallDenseMap<void *, std::pair<OwnerTy, uint64_t>, 4> UseMap;
+ /// \a Ref is the address of the \a Metadata pointer holding this. \a Owner
+ /// is notified when it changes, and is null for a direct reference such as
+ /// \a TrackingMDRef.
+ struct UseEntry {
+ void *Ref;
+ OwnerTy Owner;
+ };
+
+ /// Uses of this. Dropping one moves the last entry into its slot, so the
+ /// order is deterministic but not the order they were added in.
+ SmallVector<UseEntry, 4> UseList;
+
+ /// Ref -> index in UseList, built once UseList grows past IndexThreshold
+ /// entries and dropped once it shrinks to half that, so that a use of a
+ /// large list is found in constant time.
+ static constexpr unsigned IndexThreshold = 32;
+ std::unique_ptr<DenseMap<void *, unsigned>> Index;
protected:
~ReplaceableUses() {
- assert(UseMap.empty() && "Cannot destroy in-use replaceable metadata");
+ assert(UseList.empty() && "Cannot destroy in-use replaceable metadata");
}
public:
@@ -426,9 +441,10 @@ class ReplaceableUses {
/// is resolved.
LLVM_ABI void resolveAllUses(bool ResolveUsers = true);
- unsigned getNumUses() const { return UseMap.size(); }
+ unsigned getNumUses() const { return UseList.size(); }
private:
+ UseEntry *findRef(void *Ref);
void addRef(void *Ref, OwnerTy Owner);
void dropRef(void *Ref);
void moveRef(void *Ref, void *New, const Metadata &MD);
diff --git a/llvm/lib/IR/Metadata.cpp b/llvm/lib/IR/Metadata.cpp
index a947bcfd6fa96..85f194118bfab 100644
--- a/llvm/lib/IR/Metadata.cpp
+++ b/llvm/lib/IR/Metadata.cpp
@@ -258,82 +258,88 @@ bool MetadataTracking::isReplaceable(const Metadata &MD) {
}
SmallVector<Metadata *> ReplaceableUses::getAllArgListUsers() {
- SmallVector<std::pair<OwnerTy, uint64_t> *> MDUsersWithID;
- for (auto Pair : UseMap) {
- OwnerTy Owner = Pair.second.first;
- if (Owner.isNull())
+ SmallVector<Metadata *> MDUsers;
+ for (const UseEntry &U : UseList) {
+ if (U.Owner.isNull())
continue;
- if (!isa<Metadata *>(Owner))
+ if (!isa<Metadata *>(U.Owner))
continue;
- Metadata *OwnerMD = cast<Metadata *>(Owner);
+ Metadata *OwnerMD = cast<Metadata *>(U.Owner);
if (OwnerMD->getMetadataID() == Metadata::DIArgListKind)
- MDUsersWithID.push_back(&UseMap[Pair.first]);
+ MDUsers.push_back(OwnerMD);
}
- llvm::sort(MDUsersWithID, [](auto UserA, auto UserB) {
- return UserA->second < UserB->second;
- });
- SmallVector<Metadata *> MDUsers;
- for (auto *UserWithID : MDUsersWithID)
- MDUsers.push_back(cast<Metadata *>(UserWithID->first));
return MDUsers;
}
SmallVector<DbgVariableRecord *>
ReplaceableUses::getAllDbgVariableRecordUsers() {
- SmallVector<std::pair<OwnerTy, uint64_t> *> DVRUsersWithID;
- for (auto Pair : UseMap) {
- OwnerTy Owner = Pair.second.first;
- if (Owner.isNull())
+ // Newest first as long as no use has been dropped; see UseList.
+ SmallVector<DbgVariableRecord *> DVRUsers;
+ for (const UseEntry &U : reverse(UseList)) {
+ if (U.Owner.isNull())
continue;
- if (!isa<DebugValueUser *>(Owner))
+ if (!isa<DebugValueUser *>(U.Owner))
continue;
- DVRUsersWithID.push_back(&UseMap[Pair.first]);
- }
- // Order DbgVariableRecord users in reverse-creation order. Normal dbg.value
- // users of MetadataAsValues are ordered by their UseList, i.e. reverse order
- // of when they were added: we need to replicate that here. The structure of
- // debug-info output depends on the ordering of intrinsics, thus we need
- // to keep them consistent for comparisons sake.
- llvm::sort(DVRUsersWithID, [](auto UserA, auto UserB) {
- return UserA->second > UserB->second;
- });
- SmallVector<DbgVariableRecord *> DVRUsers;
- for (auto UserWithID : DVRUsersWithID)
- DVRUsers.push_back(cast<DebugValueUser *>(UserWithID->first)->getUser());
+ DVRUsers.push_back(cast<DebugValueUser *>(U.Owner)->getUser());
+ }
return DVRUsers;
}
-void ReplaceableUses::addRef(void *Ref, OwnerTy Owner) {
- bool WasInserted =
- UseMap.insert(std::make_pair(Ref, std::make_pair(Owner, NextIndex)))
- .second;
- (void)WasInserted;
- assert(WasInserted && "Expected to add a reference");
+ReplaceableUses::UseEntry *ReplaceableUses::findRef(void *Ref) {
+ if (Index) {
+ auto It = Index->find(Ref);
+ return It == Index->end() ? nullptr : &UseList[It->second];
+ }
+ // Scan backwards: a reference is usually dropped or moved soon after it is
+ // added.
+ for (UseEntry &U : reverse(UseList))
+ if (U.Ref == Ref)
+ return &U;
+ return nullptr;
+}
- ++NextIndex;
- assert(NextIndex != 0 && "Unexpected overflow");
+void ReplaceableUses::addRef(void *Ref, OwnerTy Owner) {
+ assert(!findRef(Ref) && "Expected to add a reference");
+ UseList.push_back({Ref, Owner});
+ if (Index) {
+ (*Index)[Ref] = UseList.size() - 1;
+ } else if (UseList.size() > IndexThreshold) {
+ Index = std::make_unique<DenseMap<void *, unsigned>>();
+ for (auto [I, U] : enumerate(UseList))
+ (*Index)[U.Ref] = I;
+ }
}
void ReplaceableUses::dropRef(void *Ref) {
- bool WasErased = UseMap.erase(Ref);
- (void)WasErased;
- assert(WasErased && "Expected to drop a reference");
+ UseEntry *U = findRef(Ref);
+ assert(U && "Expected to drop a reference");
+ if (Index)
+ Index->erase(Ref);
+ if (U != &UseList.back()) {
+ *U = UseList.back();
+ if (Index)
+ (*Index)[U->Ref] = U - UseList.begin();
+ }
+ UseList.pop_back();
+ if (Index && UseList.size() <= IndexThreshold / 2)
+ Index.reset();
}
void ReplaceableUses::moveRef(void *Ref, void *New, const Metadata &MD) {
- auto I = UseMap.find(Ref);
- assert(I != UseMap.end() && "Expected to move a reference");
- auto OwnerAndIndex = I->second;
- UseMap.erase(I);
- bool WasInserted = UseMap.insert(std::make_pair(New, OwnerAndIndex)).second;
- (void)WasInserted;
- assert(WasInserted && "Expected to add a reference");
+ assert(!findRef(New) && "Expected to add a reference");
+ UseEntry *U = findRef(Ref);
+ assert(U && "Expected to move a reference");
+ if (Index) {
+ Index->erase(Ref);
+ (*Index)[New] = U - UseList.begin();
+ }
+ U->Ref = New;
// Check that the references are direct if there's no owner.
(void)MD;
- assert((OwnerAndIndex.first || *static_cast<Metadata **>(Ref) == &MD) &&
+ assert((U->Owner || *static_cast<Metadata **>(Ref) == &MD) &&
"Reference without owner must be direct");
- assert((OwnerAndIndex.first || *static_cast<Metadata **>(New) == &MD) &&
+ assert((U->Owner || *static_cast<Metadata **>(New) == &MD) &&
"Reference without owner must be direct");
}
@@ -346,14 +352,11 @@ void ReplaceableUses::SalvageDebugInfo(const Constant &C) {
auto &Store = Context.pImpl->ValuesAsMetadata;
auto I = Store.find(&C);
ValueAsMetadata *MD = I->second;
- using UseTy =
- std::pair<void *, std::pair<MetadataTracking::OwnerTy, uint64_t>>;
// Copy out uses and update value of Constant used by debug info metadata with
// poison below
- SmallVector<UseTy, 8> Uses(MD->UseMap.begin(), MD->UseMap.end());
-
- for (const auto &Pair : Uses) {
- MetadataTracking::OwnerTy Owner = Pair.second.first;
+ SmallVector<UseEntry, 8> Uses(MD->UseList.begin(), MD->UseList.end());
+ for (const UseEntry &U : Uses) {
+ MetadataTracking::OwnerTy Owner = U.Owner;
if (!Owner)
continue;
// Check for MetadataAsValue.
@@ -369,35 +372,31 @@ void ReplaceableUses::SalvageDebugInfo(const Constant &C) {
continue;
if (isa<DINode>(OwnerMD)) {
OwnerMD->handleChangedOperand(
- Pair.first, ValueAsMetadata::get(PoisonValue::get(C.getType())));
+ U.Ref, ValueAsMetadata::get(PoisonValue::get(C.getType())));
}
}
}
void ReplaceableUses::replaceAllUsesWith(Metadata *MD) {
- if (UseMap.empty())
+ if (UseList.empty())
return;
- // Copy out uses since UseMap will get touched below.
- using UseTy = std::pair<void *, std::pair<OwnerTy, uint64_t>>;
- SmallVector<UseTy, 8> Uses(UseMap.begin(), UseMap.end());
- llvm::sort(Uses, [](const UseTy &L, const UseTy &R) {
- return L.second.second < R.second.second;
- });
- for (const auto &Pair : Uses) {
+ // Copy out uses since UseList will get touched below.
+ SmallVector<UseEntry, 8> Uses(UseList.begin(), UseList.end());
+ for (const UseEntry &U : Uses) {
// Check that this Ref hasn't disappeared after RAUW (when updating a
// previous Ref).
- if (!UseMap.count(Pair.first))
+ if (!findRef(U.Ref))
continue;
- OwnerTy Owner = Pair.second.first;
+ OwnerTy Owner = U.Owner;
if (!Owner) {
// Update unowned tracking references directly.
- Metadata *&Ref = *static_cast<Metadata **>(Pair.first);
+ Metadata *&Ref = *static_cast<Metadata **>(U.Ref);
Ref = MD;
if (MD)
MetadataTracking::track(Ref);
- UseMap.erase(Pair.first);
+ dropRef(U.Ref);
continue;
}
@@ -408,7 +407,7 @@ void ReplaceableUses::replaceAllUsesWith(Metadata *MD) {
}
if (auto *DVU = dyn_cast<DebugValueUser *>(Owner)) {
- DVU->handleChangedValue(Pair.first, MD);
+ DVU->handleChangedValue(U.Ref, MD);
continue;
}
@@ -417,34 +416,32 @@ void ReplaceableUses::replaceAllUsesWith(Metadata *MD) {
switch (OwnerMD->getMetadataID()) {
#define HANDLE_METADATA_LEAF(CLASS) \
case Metadata::CLASS##Kind: \
- cast<CLASS>(OwnerMD)->handleChangedOperand(Pair.first, MD); \
+ cast<CLASS>(OwnerMD)->handleChangedOperand(U.Ref, MD); \
continue;
#include "llvm/IR/Metadata.def"
default:
llvm_unreachable("Invalid metadata subclass");
}
}
- assert(UseMap.empty() && "Expected all uses to be replaced");
+ assert(UseList.empty() && "Expected all uses to be replaced");
}
void ReplaceableUses::resolveAllUses(bool ResolveUsers) {
- if (UseMap.empty())
+ if (UseList.empty())
return;
if (!ResolveUsers) {
- UseMap.clear();
+ UseList.clear();
+ Index.reset();
return;
}
- // Copy out uses since UseMap could get touched below.
- using UseTy = std::pair<void *, std::pair<OwnerTy, uint64_t>>;
- SmallVector<UseTy, 8> Uses(UseMap.begin(), UseMap.end());
- llvm::sort(Uses, [](const UseTy &L, const UseTy &R) {
- return L.second.second < R.second.second;
- });
- UseMap.clear();
- for (const auto &Pair : Uses) {
- auto Owner = Pair.second.first;
+ // Copy out uses since UseList could get touched below.
+ SmallVector<UseEntry, 8> Uses(UseList.begin(), UseList.end());
+ UseList.clear();
+ Index.reset();
+ for (const UseEntry &U : Uses) {
+ OwnerTy Owner = U.Owner;
if (!Owner)
continue;
if (!isa<Metadata *>(Owner))
diff --git a/llvm/unittests/IR/MetadataTest.cpp b/llvm/unittests/IR/MetadataTest.cpp
index e9af072ea800e..2867fc88c6cbc 100644
--- a/llvm/unittests/IR/MetadataTest.cpp
+++ b/llvm/unittests/IR/MetadataTest.cpp
@@ -5355,6 +5355,29 @@ TEST_F(TrackingMDRefTest, UpdatesOnDeletion) {
EXPECT_TRUE(!MD);
}
+// Once the use list is indexed, look each changed reference up again right
+// away: a later swap in dropRef would repair a stale index entry.
+TEST_F(TrackingMDRefTest, LargeUseList) {
+ auto Temp = MDTuple::getTemporary(Context, {});
+ TrackingMDRef Refs[41];
+ for (unsigned I = 0; I != 40; ++I)
+ Refs[I].reset(Temp.get());
+ Refs[0].reset();
+ Refs[0].reset(Temp.get());
+ Refs[0].reset();
+ Refs[40] = std::move(Refs[1]);
+ Refs[1].reset(Temp.get());
+ Refs[40].reset();
+ Refs[39].reset();
+ EXPECT_EQ(38u, Temp->getNumTemporaryUses());
+
+ auto *N = MDTuple::getDistinct(Context, {});
+ Temp->replaceAllUsesWith(N);
+ EXPECT_EQ(0u, Temp->getNumTemporaryUses());
+ for (unsigned I = 1; I != 39; ++I)
+ EXPECT_EQ(N, Refs[I].get()) << I;
+}
+
TEST(NamedMDNodeTest, Search) {
LLVMContext Context;
ConstantAsMetadata *C =
More information about the llvm-commits
mailing list