[llvm] b3411b8 - [FoldingSet] Switch to linear probing and Algorithm R deletion (#218190)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Aug 26 11:51:26 PDT 2026
Author: Fangrui Song
Date: 2026-08-26T18:51:20Z
New Revision: b3411b89c3afaffb0a102169065b317e7a322d96
URL: https://github.com/llvm/llvm-project/commit/b3411b89c3afaffb0a102169065b317e7a322d96
DIFF: https://github.com/llvm/llvm-project/commit/b3411b89c3afaffb0a102169065b317e7a322d96.diff
LOG: [FoldingSet] Switch to linear probing and Algorithm R deletion (#218190)
FoldingSet uses an inefficient chaining hash table. Switch to
linear-probing open addressing: the bucket array holds node pointers,
with null marking an empty slot. Deletion uses Knuth TAOCP 6.4 Algorithm
R, as DenseMap/StringMap do, so erase invalidates iterators while
leaving pointers to nodes valid.
The next-in-bucket pointer becomes a cached 32-bit hash, halving
FoldingSetNode. A probe compares it before the profile compare, and
FindNodeOrInsertPos returns it instead of a bucket address, so an
InsertPos survives intervening insertions.
https://discourse.llvm.org/t/rfc-modernizing-llvms-foldingset-open-addressing-with-swiss-table-and-algorithm-r/91637
LLM-aided
Co-authored-by: Kazu Hirata <kazu at google.com>
Added:
Modified:
llvm/docs/ProgrammersManual.md
llvm/include/llvm/ADT/FoldingSet.h
llvm/lib/Support/FoldingSet.cpp
llvm/unittests/ADT/FoldingSet.cpp
Removed:
################################################################################
diff --git a/llvm/docs/ProgrammersManual.md b/llvm/docs/ProgrammersManual.md
index 16f24d6565fad..e4c07a389891a 100644
--- a/llvm/docs/ProgrammersManual.md
+++ b/llvm/docs/ProgrammersManual.md
@@ -2058,8 +2058,8 @@ building composite data structures.
#### llvm/ADT/FoldingSet.h
`FoldingSet` is an aggregate class that is really good at uniquing
-expensive-to-create or polymorphic objects. It is a combination of a chained
-hash table with intrusive links (uniqued objects are required to inherit from
+expensive-to-create or polymorphic objects. It is a linear-probed hash table
+whose buckets point to the uniqued objects (which are required to inherit from
`FoldingSetNode`) that uses {ref}`SmallVector <dss_smallvector>` as part of its ID
process.
@@ -2076,11 +2076,12 @@ element that we want to query for. The query either returns the element
matching the ID or it returns an opaque ID that indicates where insertion should
take place. Construction of the ID usually does not require heap traffic.
-Because `FoldingSet` uses intrusive links, it can support polymorphic objects in
+Because the buckets are pointers, `FoldingSet` can support polymorphic objects in
the set (for example, you can have `SDNode` instances mixed with `LoadSDNodes`).
Because the elements are individually allocated, pointers to the elements are
stable: inserting or removing elements does not invalidate any pointers to other
-elements.
+elements. The iterators, however, are invalidated whenever an insertion or
+erasure occurs, as in `DenseMap` and `StringMap`.
(dss_set)=
diff --git a/llvm/include/llvm/ADT/FoldingSet.h b/llvm/include/llvm/ADT/FoldingSet.h
index 9b324b337072c..bc3e5bcc8ad3e 100644
--- a/llvm/include/llvm/ADT/FoldingSet.h
+++ b/llvm/include/llvm/ADT/FoldingSet.h
@@ -35,12 +35,11 @@ namespace llvm {
/// This folding set is used for two purposes:
/// 1. Given information about a node we want to create, look up the unique
/// instance of the node in the set. If the node already exists, return
-/// it, otherwise return the bucket it should be inserted into.
+/// it, otherwise return a token that makes the insertion cheap.
/// 2. Given a node that has already been created, remove it from the set.
///
-/// This class is implemented as a single-link chained hash table, where the
-/// "buckets" are actually the nodes themselves (the next pointer is in the
-/// node). The last node points back to the bucket to simplify node removal.
+/// The hash table is linear-probing open addressing with tombstone-free
+/// deletion, power-of-two capacity, and a 0.75 maximum load factor.
///
/// Any node that is to be included in the folding set must be a subclass of
/// FoldingSetNode. The node class must also define a Profile method used to
@@ -101,6 +100,9 @@ namespace llvm {
/// MyNode *N = new MyNode(Name, Value);
/// MyFoldingSet.InsertNode(N, InsertPoint);
///
+/// InsertPoint survives intervening insertions, but N must profile identically
+/// to the ID that produced it, or N becomes unfindable.
+///
/// 4) Finally, if you want to remove a node from the folding set call;
///
/// bool WasRemoved = MyFoldingSet.RemoveNode(M);
@@ -180,10 +182,16 @@ class FoldingSetNodeIDRef {
FoldingSetNodeIDRef() = default;
FoldingSetNodeIDRef(const unsigned *D, size_t S) : Data(D), Size(S) {}
+ static constexpr unsigned NotAHash = 0;
+
// Compute a strong hash value used to lookup the node in the FoldingSetBase.
// The hash value is not guaranteed to be deterministic across processes.
+ // Never returns NotAHash: FoldingSetBase uses it to keep the InsertPos token
+ // non-null and to mark a node belonging to no set.
unsigned ComputeHash() const {
- return static_cast<unsigned>(hash_combine_range(Data, Data + Size));
+ unsigned Hash =
+ static_cast<unsigned>(hash_combine_range(Data, Data + Size));
+ return Hash == NotAHash ? 1 : Hash;
}
// Compute a deterministic hash value across processes that is suitable for
@@ -290,22 +298,17 @@ class FoldingSetNodeID {
};
//===----------------------------------------------------------------------===//
-/// Implements the folding set functionality. The main structure is an array of
-/// buckets. Each bucket is indexed by the hash of the nodes it contains. The
-/// bucket itself points to the nodes contained in the bucket via a singly
-/// linked list. The last node in the list points back to the bucket to
-/// facilitate node removal.
-///
+/// Non-templated base class for FoldingSet and ContextualFoldingSet, holding
+/// the memory management and probing that does not depend on the node type.
class FoldingSetBase : public DebugEpochBase {
protected:
- /// Array of bucket chains.
+ /// Array of node pointers; a null entry marks an empty slot.
void **Buckets = nullptr;
/// Length of the Buckets array. Always a power of 2.
unsigned NumBuckets = 0;
- /// Number of nodes in the folding set. Growth occurs when NumNodes
- /// is greater than twice the number of buckets.
+ /// Number of nodes in the folding set.
unsigned NumNodes = 0;
LLVM_ABI explicit FoldingSetBase(unsigned Log2InitSize);
@@ -315,19 +318,19 @@ class FoldingSetBase : public DebugEpochBase {
public:
//===--------------------------------------------------------------------===//
- /// This class is used to maintain the singly linked bucket list in
- /// a folding set.
+ /// This class is used to maintain node state in a folding set.
class Node {
private:
- // NextInFoldingSetBucket - next link in the bucket list.
- void *NextInFoldingSetBucket = nullptr;
+ // Hash of the node's profile, cached so that growth and removal never
+ // re-run Profile(). NotAHash while the node is in no folding set.
+ uint32_t FoldingSetHash = FoldingSetNodeIDRef::NotAHash;
public:
Node() = default;
// Accessors
- void *getNextInBucket() const { return NextInFoldingSetBucket; }
- void SetNextInBucket(void *N) { NextInFoldingSetBucket = N; }
+ uint32_t getFoldingSetHash() const { return FoldingSetHash; }
+ void setFoldingSetHash(uint32_t Hash) { FoldingSetHash = Hash; }
};
/// Remove all nodes from the folding set.
@@ -339,13 +342,9 @@ class FoldingSetBase : public DebugEpochBase {
/// Returns true if there are no nodes in the folding set.
[[nodiscard]] bool empty() const { return NumNodes == 0; }
- /// Returns the number of nodes permitted in the folding set
- /// before a rebucket operation is performed.
- unsigned capacity() const {
- // We allow a load factor of up to 2.0,
- // so that means our capacity is NumBuckets * 2
- return NumBuckets * 2;
- }
+ /// Grow the number of buckets so that we can hold at least \p N nodes
+ /// before rebucketing. May allocate more space than requested.
+ LLVM_ABI void reserve(unsigned N);
protected:
/// Functions provided by the derived class to compute folding properties.
@@ -370,18 +369,25 @@ class FoldingSetBase : public DebugEpochBase {
};
private:
- /// Resize the hash table and rehash everything. \p NewBucketCount must be a
- /// power of two, and must be greater than the old bucket count.
- void GrowBucketCount(unsigned NewBucketCount, const FoldingSetInfo &Info);
+ /// Put \p N in the first empty slot following its home, without checking
+ /// capacity. Does not touch \p N, so a rehash need not dirty every node.
+ void placeNode(Node *N, uint32_t Hash);
+
+ /// Compare \p N against \p ID. Out of line to keep FoldingSetNodeID's inline
+ /// storage out of the probe loop's frame.
+ static bool nodeEquals(const FoldingSetInfo &Info, const FoldingSetBase *Self,
+ Node *N, const FoldingSetNodeID &ID, unsigned IDHash);
+
+ friend class FoldingSetIteratorImpl;
+
+ /// Rehash into at least \p MinNumBuckets buckets, rounded up to a power of
+ /// two and floored at the constructor's minimum.
+ void grow(unsigned MinNumBuckets);
protected:
// The below methods are protected to encourage subclasses to provide a more
// type-safe API.
- /// Grow the number of buckets so that we can hold at least \p EltCount
- /// nodes before rebucketing. May allocate more space than requested.
- LLVM_ABI void reserve(unsigned EltCount, const FoldingSetInfo &Info);
-
/// Remove a node from the folding set, returning true if one
/// was removed or false if the node was not in the folding set.
LLVM_ABI bool RemoveNode(Node *N);
@@ -398,9 +404,8 @@ class FoldingSetBase : public DebugEpochBase {
/// Insert the specified node into the folding set, knowing that
/// it is not already in the folding set. InsertPos must be obtained from
- /// FindNodeOrInsertPos.
- LLVM_ABI void InsertNode(Node *N, void *InsertPos,
- const FoldingSetInfo &Info);
+ /// FindNodeOrInsertPos for an ID that \p N profiles identically to.
+ LLVM_ABI void InsertNode(Node *N, void *InsertPos);
};
// Convenience type to hide the implementation of the folding set.
@@ -497,21 +502,13 @@ class FoldingSetImpl : public FoldingSetBase, public Trait::ContextStorage {
public:
using iterator = FoldingSetIterator<T>;
- iterator begin() { return iterator(this, Buckets); }
- iterator end() { return iterator(this, Buckets + NumBuckets); }
+ iterator begin() { return iterator(this, 0); }
+ iterator end() { return iterator(this, NumBuckets); }
using const_iterator = FoldingSetIterator<const T>;
- const_iterator begin() const { return const_iterator(this, Buckets); }
- const_iterator end() const {
- return const_iterator(this, Buckets + NumBuckets);
- }
-
- /// Grow the number of buckets so that we can hold at least \p EltCount
- /// nodes before rebucketing. May allocate more space than requested.
- void reserve(unsigned EltCount) {
- FoldingSetBase::reserve(EltCount, getFoldingSetInfo());
- }
+ const_iterator begin() const { return const_iterator(this, 0); }
+ const_iterator end() const { return const_iterator(this, NumBuckets); }
/// Remove a node from the folding set, returning true if one
/// was removed or false if the node was not in the folding set.
@@ -535,7 +532,7 @@ class FoldingSetImpl : public FoldingSetBase, public Trait::ContextStorage {
/// it is not already in the folding set. InsertPos must be obtained from
/// FindNodeOrInsertPos.
void InsertNode(T *N, void *InsertPos) {
- FoldingSetBase::InsertNode(N, InsertPos, getFoldingSetInfo());
+ FoldingSetBase::InsertNode(N, InsertPos);
}
/// Insert the specified node into the folding set, knowing that it is not
@@ -641,21 +638,22 @@ template <class T, class VectorT = SmallVector<T *, 8>> class FoldingSetVector {
/// how to walk the folding set hash table.
class FoldingSetIteratorImpl : DebugEpochBase::HandleBase {
protected:
- FoldingSetNode *NodePtr;
+ const FoldingSetBase *Set = nullptr;
+ unsigned Index = 0;
- LLVM_ABI FoldingSetIteratorImpl(const DebugEpochBase *Epoch, void **Bucket);
+ LLVM_ABI FoldingSetIteratorImpl(const FoldingSetBase *Set, unsigned Index);
LLVM_ABI void advance();
FoldingSetNode *getNode() const {
assert(isHandleInSync() && "invalid iterator access!");
- return NodePtr;
+ return static_cast<FoldingSetNode *>(Set->Buckets[Index]);
}
public:
bool operator==(const FoldingSetIteratorImpl &RHS) const {
assert(isHandleInSync() && RHS.isHandleInSync() && "handle not in sync!");
- return NodePtr == RHS.NodePtr;
+ return Set == RHS.Set && Index == RHS.Index;
}
bool operator!=(const FoldingSetIteratorImpl &RHS) const {
return !(*this == RHS);
@@ -664,8 +662,8 @@ class FoldingSetIteratorImpl : DebugEpochBase::HandleBase {
template <class T> class FoldingSetIterator : public FoldingSetIteratorImpl {
public:
- explicit FoldingSetIterator(const DebugEpochBase *Epoch, void **Bucket)
- : FoldingSetIteratorImpl(Epoch, Bucket) {}
+ explicit FoldingSetIterator(const FoldingSetBase *Set, unsigned Index)
+ : FoldingSetIteratorImpl(Set, Index) {}
T &operator*() const { return *static_cast<T *>(getNode()); }
diff --git a/llvm/lib/Support/FoldingSet.cpp b/llvm/lib/Support/FoldingSet.cpp
index 13aac74280978..fc5da49c3ff47 100644
--- a/llvm/lib/Support/FoldingSet.cpp
+++ b/llvm/lib/Support/FoldingSet.cpp
@@ -131,56 +131,22 @@ FoldingSetNodeID::Intern(BumpPtrAllocator &Allocator) const {
}
//===----------------------------------------------------------------------===//
-/// Helper functions for FoldingSetBase.
-
-/// GetNextPtr - In order to save space, each bucket is a
-/// singly-linked-list. In order to make deletion more efficient, we make
-/// the list circular, so we can delete a node without computing its hash.
-/// The problem with this is that the start of the hash buckets are not
-/// Nodes. If NextInBucketPtr is a bucket pointer, this method returns null:
-/// use GetBucketPtr when this happens.
-static FoldingSetBase::Node *GetNextPtr(void *NextInBucketPtr) {
- // The low bit is set if this is the pointer back to the bucket.
- if (reinterpret_cast<intptr_t>(NextInBucketPtr) & 1)
- return nullptr;
-
- return static_cast<FoldingSetBase::Node *>(NextInBucketPtr);
-}
-
-/// GetBucketPtr - Provides a casting of a bucket pointer for isNode
-/// testing.
-static void **GetBucketPtr(void *NextInBucketPtr) {
- intptr_t Ptr = reinterpret_cast<intptr_t>(NextInBucketPtr);
- assert((Ptr & 1) && "Not a bucket pointer");
- return reinterpret_cast<void **>(Ptr & ~intptr_t(1));
-}
+// FoldingSetBase Implementation
-/// GetBucketFor - Hash the specified node ID and return the hash bucket for
-/// the specified ID.
-static void **GetBucketFor(unsigned Hash, void **Buckets, unsigned NumBuckets) {
- // NumBuckets is always a power of 2.
- unsigned BucketNum = Hash & (NumBuckets - 1);
- return Buckets + BucketNum;
+/// Encode a 32-bit hash as an opaque non-null token for InsertPos.
+static void *encodeHash(uint32_t Hash) {
+ return reinterpret_cast<void *>(static_cast<uintptr_t>(Hash));
}
-/// AllocateBuckets - Allocate initialized bucket memory.
-static void **AllocateBuckets(unsigned NumBuckets) {
- void **Buckets =
- static_cast<void **>(safe_calloc(NumBuckets + 1, sizeof(void *)));
- // Set the very last bucket to be a non-null "pointer".
- Buckets[NumBuckets] = reinterpret_cast<void *>(-1);
- return Buckets;
+static uint32_t decodeHash(void *InsertPos) {
+ return static_cast<uint32_t>(reinterpret_cast<uintptr_t>(InsertPos));
}
-//===----------------------------------------------------------------------===//
-// FoldingSetBase Implementation
-
FoldingSetBase::FoldingSetBase(unsigned Log2InitSize) {
assert(5 < Log2InitSize && Log2InitSize < 32 &&
"Initial hash table size out of range");
NumBuckets = 1 << Log2InitSize;
- Buckets = AllocateBuckets(NumBuckets);
- NumNodes = 0;
+ Buckets = static_cast<void **>(safe_calloc(NumBuckets, sizeof(void *)));
}
FoldingSetBase::FoldingSetBase(FoldingSetBase &&Arg)
@@ -207,145 +173,108 @@ FoldingSetBase::~FoldingSetBase() { free(Buckets); }
void FoldingSetBase::clear() {
incrementEpoch();
- // Set all but the last bucket to null pointers.
- memset(Buckets, 0, NumBuckets * sizeof(void *));
-
- // Set the very last bucket to be a non-null "pointer".
- Buckets[NumBuckets] = reinterpret_cast<void *>(-1);
-
- // Reset the node count to zero.
+ // Stale hashes are unreachable, so only the occupancy needs resetting.
+ if (NumBuckets)
+ memset(Buckets, 0, NumBuckets * sizeof(void *));
NumNodes = 0;
}
-void FoldingSetBase::GrowBucketCount(unsigned NewBucketCount,
- const FoldingSetInfo &Info) {
- assert((NewBucketCount > NumBuckets) &&
- "Can't shrink a folding set with GrowBucketCount");
- assert(isPowerOf2_32(NewBucketCount) && "Bad bucket count!");
+void FoldingSetBase::placeNode(Node *N, uint32_t Hash) {
+ unsigned Mask = NumBuckets - 1;
+ unsigned I = Hash & Mask;
+ while (Buckets[I]) {
+ assert(Buckets[I] != N && "Node already in the folding set");
+ I = (I + 1) & Mask;
+ }
+ Buckets[I] = N;
+ ++NumNodes;
+}
+
+void FoldingSetBase::grow(unsigned MinNumBuckets) {
+ // The floor is the smallest size the constructor accepts.
+ unsigned NewBucketCount = std::max(64u, llvm::bit_ceil(MinNumBuckets));
+ assert(NewBucketCount > NumBuckets && "Can't shrink a folding set");
FoldingSetBase Tmp(llvm::Log2_32(NewBucketCount));
- FoldingSetNodeID TempID;
- for (unsigned i = 0; i != NumBuckets; ++i) {
- void *Probe = Buckets[i];
- if (!Probe)
- continue;
- while (Node *NodeInBucket = GetNextPtr(Probe)) {
- // Figure out the next link, remove NodeInBucket from the old link.
- Probe = NodeInBucket->getNextInBucket();
- NodeInBucket->SetNextInBucket(nullptr);
-
- // Insert the node into the new bucket, after recomputing the hash.
- Tmp.InsertNode(
- NodeInBucket,
- GetBucketFor(Info.ComputeNodeHash(this, NodeInBucket, TempID),
- Tmp.Buckets, Tmp.NumBuckets),
- Info);
- TempID.clear();
- }
- }
+ for (unsigned I = 0; I != NumBuckets; ++I)
+ if (void *N = Buckets[I])
+ Tmp.placeNode(static_cast<Node *>(N),
+ static_cast<Node *>(N)->getFoldingSetHash());
*this = std::move(Tmp);
}
-void FoldingSetBase::reserve(unsigned EltCount, const FoldingSetInfo &Info) {
- // This will give us somewhere between EltCount / 2 and
- // EltCount buckets. This puts us in the load factor
- // range of 1.0 - 2.0.
- if (EltCount <= capacity())
+void FoldingSetBase::reserve(unsigned N) {
+ if (N * 4 <= NumBuckets * 3)
return;
- GrowBucketCount(llvm::bit_floor(EltCount), Info);
+ // N + (N + 2) / 3 is ceil(4N/3).
+ grow(N + (N + 2) / 3);
+}
+
+LLVM_ATTRIBUTE_NOINLINE bool
+FoldingSetBase::nodeEquals(const FoldingSetInfo &Info,
+ const FoldingSetBase *Self, Node *N,
+ const FoldingSetNodeID &ID, unsigned IDHash) {
+ FoldingSetNodeID TempID;
+ return Info.NodeEquals(Self, N, ID, IDHash, TempID);
}
FoldingSetBase::Node *FoldingSetBase::FindNodeOrInsertPos(
const FoldingSetNodeID &ID, void *&InsertPos, const FoldingSetInfo &Info) {
unsigned IDHash = ID.ComputeHash();
- void **Bucket = GetBucketFor(IDHash, Buckets, NumBuckets);
- void *Probe = *Bucket;
-
- InsertPos = nullptr;
-
- FoldingSetNodeID TempID;
- while (Node *NodeInBucket = GetNextPtr(Probe)) {
- if (Info.NodeEquals(this, NodeInBucket, ID, IDHash, TempID))
- return NodeInBucket;
- TempID.clear();
-
- Probe = NodeInBucket->getNextInBucket();
+ unsigned Mask = NumBuckets - 1;
+ for (unsigned I = IDHash & Mask; Buckets[I]; I = (I + 1) & Mask) {
+ Node *N = static_cast<Node *>(Buckets[I]);
+ if (N->getFoldingSetHash() == IDHash &&
+ nodeEquals(Info, this, N, ID, IDHash)) {
+ InsertPos = nullptr;
+ return N;
+ }
}
- // Didn't find the node, return null with the bucket as the InsertPos.
- InsertPos = Bucket;
+ InsertPos = encodeHash(IDHash);
return nullptr;
}
-void FoldingSetBase::InsertNode(Node *N, void *InsertPos,
- const FoldingSetInfo &Info) {
- assert(!N->getNextInBucket());
+void FoldingSetBase::InsertNode(Node *N, void *InsertPos) {
+ assert(N && "Cannot insert a null node");
+ assert(InsertPos && "Invalid InsertPos!");
incrementEpoch();
- // Do we need to grow the hashtable?
- if (NumNodes + 1 > capacity()) {
- GrowBucketCount(NumBuckets * 2, Info);
- FoldingSetNodeID TempID;
- InsertPos = GetBucketFor(Info.ComputeNodeHash(this, N, TempID), Buckets,
- NumBuckets);
- }
-
- ++NumNodes;
-
- /// The insert position is actually a bucket pointer.
- void **Bucket = static_cast<void **>(InsertPos);
-
- void *Next = *Bucket;
-
- // If this is the first insertion into this bucket, its next pointer will be
- // null. Pretend as if it pointed to itself, setting the low bit to indicate
- // that it is a pointer to the bucket.
- if (!Next)
- Next = reinterpret_cast<void *>(reinterpret_cast<intptr_t>(Bucket) | 1);
-
- // Set the node's next pointer, and make the bucket point to the node.
- N->SetNextInBucket(Next);
- *Bucket = N;
+ if (LLVM_UNLIKELY((NumNodes + 1) * 4 > NumBuckets * 3))
+ grow(NumBuckets * 2);
+ uint32_t Hash = decodeHash(InsertPos);
+ placeNode(N, Hash);
+ N->setFoldingSetHash(Hash);
}
bool FoldingSetBase::RemoveNode(Node *N) {
- // Because each bucket is a circular list, we don't need to compute N's hash
- // to remove it.
- void *Ptr = N->getNextInBucket();
- if (!Ptr)
- return false; // Not in folding set.
+ uint32_t Hash = N->getFoldingSetHash();
+ if (Hash == FoldingSetNodeIDRef::NotAHash)
+ return false; // Never inserted.
+
+ unsigned Mask = NumBuckets - 1;
+ unsigned I = Hash & Mask;
+ while (Buckets[I] != N) {
+ if (LLVM_UNLIKELY(!Buckets[I]))
+ return false; // Not in folding set.
+ I = (I + 1) & Mask;
+ }
incrementEpoch();
- --NumNodes;
- N->SetNextInBucket(nullptr);
-
- // Remember what N originally pointed to, either a bucket or another node.
- void *NodeNextPtr = Ptr;
-
- // Chase around the list until we find the node (or bucket) which points to N.
- while (true) {
- if (Node *NodeInBucket = GetNextPtr(Ptr)) {
- // Advance pointer.
- Ptr = NodeInBucket->getNextInBucket();
-
- // We found a node that points to N, change it to point to N's next node,
- // removing N from the list.
- if (Ptr == N) {
- NodeInBucket->SetNextInBucket(NodeNextPtr);
- return true;
- }
- } else {
- void **Bucket = GetBucketPtr(Ptr);
- Ptr = *Bucket;
-
- // If we found that the bucket points to N, update the bucket to point to
- // whatever is next.
- if (Ptr == N) {
- *Bucket = NodeNextPtr;
- return true;
- }
+
+ // Knuth TAOCP 6.4 Algorithm R: walk forward sliding each following entry
+ // whose probe path crosses the hole.
+ for (unsigned J = (I + 1) & Mask; Buckets[J]; J = (J + 1) & Mask) {
+ unsigned Ideal = static_cast<Node *>(Buckets[J])->getFoldingSetHash();
+ if (((I - Ideal) & Mask) < ((J - Ideal) & Mask)) {
+ Buckets[I] = Buckets[J];
+ I = J;
}
}
+ Buckets[I] = nullptr;
+ N->setFoldingSetHash(FoldingSetNodeIDRef::NotAHash);
+ --NumNodes;
+ return true;
}
FoldingSetBase::Node *
@@ -355,41 +284,23 @@ FoldingSetBase::GetOrInsertNode(Node *N, const FoldingSetInfo &Info) {
void *IP;
if (Node *E = FindNodeOrInsertPos(ID, IP, Info))
return E;
- InsertNode(N, IP, Info);
+ InsertNode(N, IP);
return N;
}
//===----------------------------------------------------------------------===//
// FoldingSetIteratorImpl Implementation
-FoldingSetIteratorImpl::FoldingSetIteratorImpl(const DebugEpochBase *Epoch,
- void **Bucket)
- : DebugEpochBase::HandleBase(Epoch) {
- // Skip to the first non-null non-self-cycle bucket.
- while (*Bucket != reinterpret_cast<void *>(-1) &&
- (!*Bucket || !GetNextPtr(*Bucket)))
- ++Bucket;
-
- NodePtr = static_cast<FoldingSetNode *>(*Bucket);
+FoldingSetIteratorImpl::FoldingSetIteratorImpl(const FoldingSetBase *Set,
+ unsigned Index)
+ : DebugEpochBase::HandleBase(Set), Set(Set), Index(Index) {
+ while (this->Index < Set->NumBuckets && !Set->Buckets[this->Index])
+ ++this->Index;
}
void FoldingSetIteratorImpl::advance() {
assert(isHandleInSync() && "invalid iterator access!");
- // If there is another link within this bucket, go to it.
- void *Probe = NodePtr->getNextInBucket();
-
- if (FoldingSetNode *NextNodeInBucket = GetNextPtr(Probe))
- NodePtr = NextNodeInBucket;
- else {
- // Otherwise, this is the last link in this bucket.
- void **Bucket = GetBucketPtr(Probe);
-
- // Skip to the next non-null non-self-cycle bucket.
- do {
- ++Bucket;
- } while (*Bucket != reinterpret_cast<void *>(-1) &&
- (!*Bucket || !GetNextPtr(*Bucket)));
-
- NodePtr = static_cast<FoldingSetNode *>(*Bucket);
- }
+ do
+ ++Index;
+ while (Index < Set->NumBuckets && !Set->Buckets[Index]);
}
diff --git a/llvm/unittests/ADT/FoldingSet.cpp b/llvm/unittests/ADT/FoldingSet.cpp
index 168443b21a4ef..ba0b7326e26d4 100644
--- a/llvm/unittests/ADT/FoldingSet.cpp
+++ b/llvm/unittests/ADT/FoldingSet.cpp
@@ -13,7 +13,12 @@
#include "llvm/ADT/FoldingSet.h"
#include "gmock/gmock.h"
#include "gtest/gtest.h"
+#include <map>
+#include <memory>
+#include <random>
+#include <set>
#include <string>
+#include <vector>
using namespace llvm;
using testing::ElementsAre;
@@ -183,25 +188,27 @@ TEST(FoldingSetTest, ClearOnNonEmpty) {
EXPECT_TRUE(Trivial.empty());
}
-TEST(FoldingSetTest, CapacityLargerThanReserve) {
- FoldingSet<TrivialPair> Trivial;
- unsigned OldCapacity = Trivial.capacity();
- Trivial.reserve(OldCapacity + 1);
- EXPECT_GE(Trivial.capacity(), OldCapacity + 1);
-}
-
-TEST(FoldingSetTest, SmallReserveChangesNothing) {
- FoldingSet<TrivialPair> Trivial;
- unsigned OldCapacity = Trivial.capacity();
- Trivial.reserve(OldCapacity - 1);
- EXPECT_EQ(Trivial.capacity(), OldCapacity);
-}
+// 48 is the most the default 64 buckets hold; 49 is one past it.
+TEST(FoldingSetTest, Reserve) {
+ for (unsigned Size : {0u, 1u, 2u, 48u, 49u}) {
+ FoldingSet<TrivialPair> Set;
+ Set.reserve(Size);
-TEST(FoldingSetTest, ReserveExactCapacity) {
- FoldingSet<TrivialPair> Trivial;
- unsigned OldCapacity = Trivial.capacity();
- Trivial.reserve(OldCapacity);
- EXPECT_EQ(Trivial.capacity(), OldCapacity);
+ std::vector<std::unique_ptr<TrivialPair>> Nodes;
+ for (unsigned I = 0; I != Size; ++I) {
+ Nodes.push_back(std::make_unique<TrivialPair>(I, I));
+ Set.InsertNode(Nodes.back().get());
+ }
+ ASSERT_EQ(Size, Set.size());
+
+ for (unsigned I = 0; I != Size; ++I) {
+ FoldingSetNodeID ID;
+ ID.AddInteger(I);
+ ID.AddInteger(I);
+ void *InsertPos = nullptr;
+ EXPECT_EQ(Nodes[I].get(), Set.FindNodeOrInsertPos(ID, InsertPos));
+ }
+ }
}
TEST(FoldingSetTest, MoveConstructor) {
@@ -375,6 +382,50 @@ TEST(FoldingSetTest, SelfMoveAssignment) {
EXPECT_FALSE(Set.empty());
}
+// Exercise growth and Algorithm R shifting against a reference model.
+TEST(FoldingSetTest, InsertEraseStress) {
+ FoldingSet<TrivialPair> Set;
+ std::map<unsigned, std::unique_ptr<TrivialPair>> Model;
+ std::mt19937 Rng(42);
+ for (unsigned Op = 0; Op != 1000; ++Op) {
+ unsigned Key = Rng() % 4096;
+ FoldingSetNodeID ID;
+ ID.AddInteger(Key);
+ ID.AddInteger(Key);
+
+ auto It = Model.find(Key);
+ if (Rng() & 1) {
+ void *InsertPos = nullptr;
+ TrivialPair *Found = Set.FindNodeOrInsertPos(ID, InsertPos);
+ if (It != Model.end()) {
+ ASSERT_EQ(It->second.get(), Found);
+ continue;
+ }
+ ASSERT_EQ(nullptr, Found);
+ auto N = std::make_unique<TrivialPair>(Key, Key);
+ Set.InsertNode(N.get(), InsertPos);
+ Model.emplace(Key, std::move(N));
+ } else if (It != Model.end()) {
+ ASSERT_TRUE(Set.RemoveNode(It->second.get()));
+ ASSERT_FALSE(Set.RemoveNode(It->second.get()));
+ Model.erase(It);
+ }
+ ASSERT_EQ(Model.size(), Set.size());
+ }
+
+ for (const auto &KV : Model) {
+ FoldingSetNodeID ID;
+ ID.AddInteger(KV.first);
+ ID.AddInteger(KV.first);
+ void *InsertPos = nullptr;
+ EXPECT_EQ(KV.second.get(), Set.FindNodeOrInsertPos(ID, InsertPos));
+ }
+ std::set<TrivialPair *> Visited;
+ for (TrivialPair &N : Set)
+ EXPECT_TRUE(Visited.insert(&N).second);
+ EXPECT_EQ(Model.size(), Visited.size());
+}
+
#if LLVM_ENABLE_ABI_BREAKING_CHECKS
TEST(FoldingSetTest, InsertInvalidatesIterators) {
FoldingSet<TrivialPair> Set;
@@ -423,4 +474,28 @@ TEST(FoldingSetTest, MoveInvalidatesIterators) {
}
#endif
+// The InsertPos token is a hash, not a position, so a rehash cannot stale it.
+TEST(FoldingSetTest, InsertPosSurvivesGrowth) {
+ FoldingSet<TrivialPair> Set;
+ TrivialPair Late(9999, 9999);
+
+ FoldingSetNodeID ID;
+ Late.Profile(ID);
+ void *InsertPos = nullptr;
+ ASSERT_EQ(nullptr, Set.FindNodeOrInsertPos(ID, InsertPos));
+ ASSERT_NE(nullptr, InsertPos);
+
+ // Force several rehashes while the token is held.
+ std::vector<std::unique_ptr<TrivialPair>> Nodes;
+ for (unsigned I = 0; I != 200; ++I) {
+ Nodes.push_back(std::make_unique<TrivialPair>(I, I));
+ Set.InsertNode(Nodes.back().get());
+ }
+
+ Set.InsertNode(&Late, InsertPos);
+ void *Unused = nullptr;
+ EXPECT_EQ(&Late, Set.FindNodeOrInsertPos(ID, Unused));
+ EXPECT_EQ(201u, Set.size());
+}
+
} // namespace
More information about the llvm-commits
mailing list