[llvm] [AssumptionCache] Bound the cost of many assumptions about one value (PR #217525)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Aug 19 21:12:58 PDT 2026
https://github.com/khaki3 updated https://github.com/llvm/llvm-project/pull/217525
>From 97f6a44cb48694ec1c5168ebb4c506379d8bb197 Mon Sep 17 00:00:00 2001
From: Kazuaki Matsumura <kmatsumura at nvidia.com>
Date: Wed, 19 Aug 2026 21:12:47 -0700
Subject: [PATCH] [AssumptionCache] Limit the number of assumptions inspected
per value
Example:
```llvm
loop.5:
%iv.5 = phi i64 [ %next.5, %body.5 ], [ 0, %body.4 ]
%value.5 = phi i32 [ %arg, %body.5 ], [ 0, %body.4 ]
%cmp.5 = icmp slt i64 %iv.5, 10
br i1 %cmp.5, label %body.5, label %loop.4.latch
body.5:
%negative = icmp slt i32 %value.5, 0
call void @llvm.assume(i1 %negative)
%next.5 = add i64 %iv.5, 1
br label %loop.5
```
In this code, fully unrolling the enclosing nest clones the assumption
once per iteration, and every clone constrains `%arg`, leaving 10,009
assumptions on one value. `registerAssumption()` rescans the whole cache
on each registration, and `assumptionsFor()` returns every assumption
affecting a value, all of which its callers inspect
(`computeKnownBitsFromContext()` merges each fact). Registering n
assumptions therefore costs O(n^2), and each of the 33,300 queries here
costs O(n): `opt -O2` takes 59.6s on an asserts build, 99.5% of it in
`LoopFullUnrollPass`, against 0.4s for the same IR without the
assumption. The rescan is asserts-only; the query cost is in every
build.
Fix: bound both walks. `registerAssumption()` verifies only while the
cache holds at most 64 assumptions, the case its own comment describes;
larger caches are left to `AssumptionCacheTracker::verifyAnalysis()`,
which now also rejects a cached assumption belonging to another function
or not a call to `@llvm.assume`, and scopes its set of cached
assumptions per function. `assumptionsFor()` returns only the first
`-max-assumes-per-value` assumptions affecting a value, 1024 by default,
so a query costs at most that many visits. Callers that must see every
assumption use the new `allAssumptionsFor()`; the only one is the cache
verification in `CodeExtractor`. The limit is a global written through
`cl::location`, so `assumptionsFor()` stays inline in the header.
The oldest are kept because an assumption is only usable at contexts
below it, so those apply at the most program points, and rejecting one
is dearer than using it: a hit is a `comesBefore()` check and ends the
walk, while a miss scans up to 15 instructions for anything that could
interrupt control flow. Keeping the newest inverts both. On a value with
8192 assumptions and 2,000 queries it costs 7.35s against 0.36s, and on
8 clones of one assumption a limit of 1 leaves 5 of them where keeping
the oldest leaves 1, as no limit at all does.
Deduplicating instead is not available here. The cache is an analysis
and cannot erase the calls, and dropping entries for calls still in
the IR would contradict its own verifier. Erasing them is what already
happens, and it is the cost being bounded: assume k is known to be
redundant only because `computeKnownBits()` merged the k-1 already
cached, so removing n duplicates is itself O(n^2). No cheaper test is
available, as the clones are distinct `icmp`s, and `early-cse` and
`gvn` leave the calls alone because `@llvm.assume` writes inaccessible
memory. Nor are duplicates the general case: the six assumptions in
`Analysis/ScalarEvolution/avoid-assume-hang.ll` are all different. The
cleanup outlives the limit, as the assumption which proves the copies
redundant is the first one.
The example takes 3.7s and its `-O2` output is unchanged. Normal code is
unaffected: below the limit `assumptionsFor()` returns what it returned
before, so no fact is lost, and real code stays far below it. Of the
45,340 `.ll` files under `llvm/test`, only the file above exceeds even a
limit of 16, and it exists because of this same explosion; the C++ and
Fortran translation units tested exceed none. `opt -O3` over
pre-optimization IR from six LLVM translation units shows no difference
beyond run-to-run noise.
<details>
<summary>Assumption-count scaling</summary>
`opt -passes=instsimplify` over a function making 2,000 queries against
a value affected by a varying number of assumptions:
```
assumptions unpatched patched
64 0.03s 0.03s
256 0.04s 0.04s
1024 0.21s 0.21s
2048 0.70s 0.23s
4096 2.55s 0.27s
8192 9.86s 0.36s
```
This bounds the constant, not the shape: a value with 1024 assumptions
is still 1024 visits per query.
</details>
<details>
<summary>Full reproducer</summary>
```llvm
define void @nested(i32 %arg) {
entry:
br label %loop.1
loop.1:
%iv.1 = phi i64 [ %next.1, %loop.1.latch ], [ 0, %entry ]
%cmp.1 = icmp slt i64 %iv.1, 10
br i1 %cmp.1, label %loop.2, label %exit
loop.2:
%iv.2 = phi i64 [ %next.2, %loop.2.latch ], [ 0, %loop.1 ]
%value.2 = phi i32 [ %next.value.2, %loop.2.latch ], [ 0, %loop.1 ]
%cmp.2 = icmp slt i64 %iv.2, 10
br i1 %cmp.2, label %body.2, label %loop.1.latch
body.2:
%next.value.2 = add i32 %value.2, 1
%enter.3 = icmp slt i32 %value.2, 0
br i1 %enter.3, label %loop.3, label %loop.2.latch
loop.3:
%iv.3 = phi i64 [ %next.3, %loop.3.latch ], [ 0, %body.2 ]
%cmp.3 = icmp slt i64 %iv.3, 10
br i1 %cmp.3, label %loop.4, label %loop.2.latch
loop.4:
%iv.4 = phi i64 [ %next.4, %loop.4.latch ], [ 0, %loop.3 ]
%value.4 = phi i32 [ %next.value.4, %loop.4.latch ], [ 0, %loop.3 ]
%cmp.4 = icmp slt i64 %iv.4, 10
br i1 %cmp.4, label %body.4, label %loop.3.latch
body.4:
%next.value.4 = add i32 %value.4, 1
%enter.5 = icmp slt i32 %value.4, 0
br i1 %enter.5, label %loop.5, label %loop.4.latch
loop.5:
%iv.5 = phi i64 [ %next.5, %body.5 ], [ 0, %body.4 ]
%value.5 = phi i32 [ %arg, %body.5 ], [ 0, %body.4 ]
%cmp.5 = icmp slt i64 %iv.5, 10
br i1 %cmp.5, label %body.5, label %loop.4.latch
body.5:
%negative = icmp slt i32 %value.5, 0
call void @llvm.assume(i1 %negative)
call void @llvm.stackrestore.p0(ptr null)
%next.5 = add i64 %iv.5, 1
br label %loop.5
loop.4.latch:
%next.4 = add i64 %iv.4, 1
br label %loop.4
loop.3.latch:
%next.3 = add i64 %iv.3, 1
br label %loop.3
loop.2.latch:
%next.2 = add i64 %iv.2, 1
br label %loop.2
loop.1.latch:
%next.1 = add i64 %iv.1, 1
br label %loop.1
exit:
ret void
}
declare void @llvm.stackrestore.p0(ptr) #0
declare void @llvm.assume(i1 noundef) #1
attributes #0 = { nocallback nofree nosync nounwind willreturn }
attributes #1 = {
nocallback nofree nosync nounwind willreturn
memory(inaccessiblemem: write)
}
```
</details>
---
llvm/include/llvm/Analysis/AssumptionCache.h | 19 ++++++
llvm/lib/Analysis/AssumptionCache.cpp | 63 +++++++++++++------
llvm/lib/Transforms/Utils/CodeExtractor.cpp | 2 +-
.../AssumptionCache/max-assumes-per-value.ll | 37 +++++++++++
4 files changed, 102 insertions(+), 19 deletions(-)
create mode 100644 llvm/test/Analysis/AssumptionCache/max-assumes-per-value.ll
diff --git a/llvm/include/llvm/Analysis/AssumptionCache.h b/llvm/include/llvm/Analysis/AssumptionCache.h
index ed335c084fde8..08bf7659a944f 100644
--- a/llvm/include/llvm/Analysis/AssumptionCache.h
+++ b/llvm/include/llvm/Analysis/AssumptionCache.h
@@ -34,6 +34,9 @@ class raw_ostream;
class TargetTransformInfo;
class Value;
+/// Set by -max-assumes-per-value; see AssumptionCache::assumptionsFor().
+LLVM_ABI extern unsigned MaxAssumesPerValue;
+
/// A cache of \@llvm.assume calls within a function.
///
/// This cache provides fast lookup of assumptions within a function by caching
@@ -163,7 +166,23 @@ class AssumptionCache {
}
/// Access the list of assumptions which affect this value.
+ ///
+ /// Callers inspect every assumption returned, so this returns only the first
+ /// -max-assumes-per-value of them.
MutableArrayRef<ResultElem> assumptionsFor(const Value *V) {
+ MutableArrayRef<ResultElem> Assumptions = allAssumptionsFor(V);
+ if (Assumptions.size() > MaxAssumesPerValue)
+ return Assumptions.take_front(MaxAssumesPerValue);
+
+ return Assumptions;
+ }
+
+ /// Access the list of assumptions which affect this value, ignoring the
+ /// -max-assumes-per-value limit.
+ ///
+ /// Only for callers which must observe every assumption, such as cache
+ /// verification. Analyses should use assumptionsFor().
+ MutableArrayRef<ResultElem> allAssumptionsFor(const Value *V) {
if (!Scanned)
scanFunction();
diff --git a/llvm/lib/Analysis/AssumptionCache.cpp b/llvm/lib/Analysis/AssumptionCache.cpp
index 87e63e95a46d6..fcf51c1414699 100644
--- a/llvm/lib/Analysis/AssumptionCache.cpp
+++ b/llvm/lib/Analysis/AssumptionCache.cpp
@@ -43,6 +43,14 @@ static cl::opt<bool>
cl::desc("Enable verification of assumption cache"),
cl::init(false));
+unsigned llvm::MaxAssumesPerValue = 1024;
+
+static cl::opt<unsigned, true> MaxAssumesPerValueOpt(
+ "max-assumes-per-value", cl::Hidden, cl::location(MaxAssumesPerValue),
+ cl::init(1024),
+ cl::desc("Maximum number of assumptions affecting a single value that "
+ "analyses will inspect"));
+
SmallVector<AssumptionCache::ResultElem, 1> &
AssumptionCache::getOrInsertAffectedValues(Value *V) {
// Try using find_as first to avoid creating extra value handles just for the
@@ -209,6 +217,27 @@ void AssumptionCache::scanFunction() {
updateAffectedValues(cast<AssumeInst>(A));
}
+/// Check the assumptions cached for \p F, collecting them in \p Cached. Returns
+/// a description of the first invariant violated, or nullptr if there is none.
+static const char *
+findCacheViolation(const Function &F, ArrayRef<WeakVH> Assumptions,
+ SmallPtrSetImpl<const CallInst *> &Cached) {
+ for (const WeakVH &VH : Assumptions) {
+ if (!VH)
+ continue;
+
+ const auto *CI = cast<CallInst>(VH);
+ if (CI->getFunction() != &F)
+ return "Cached assumption not inside this function";
+ if (!match(CI, m_Intrinsic<Intrinsic::assume>()))
+ return "Cached something other than a call to @llvm.assume";
+ if (!Cached.insert(CI).second)
+ return "Cache contains multiple copies of a call";
+ }
+
+ return nullptr;
+}
+
void AssumptionCache::registerAssumption(AssumeInst *CI) {
// If we haven't scanned the function yet, just drop this assumption. It will
// be found when we scan later.
@@ -225,18 +254,14 @@ void AssumptionCache::registerAssumption(AssumeInst *CI) {
// We expect the number of assumptions to be small, so in an asserts build
// check that we don't accumulate duplicates and that all assumptions point
- // to the same function.
- SmallPtrSet<Value *, 16> AssumptionSet;
- for (auto &VH : AssumeHandles) {
- if (!VH)
- continue;
-
- assert(&F == cast<Instruction>(VH)->getParent()->getParent() &&
- "Cached assumption not inside this function!");
- assert(match(cast<CallInst>(VH), m_Intrinsic<Intrinsic::assume>()) &&
- "Cached something other than a call to @llvm.assume!");
- assert(AssumptionSet.insert(VH).second &&
- "Cache contains multiple copies of a call!");
+ // to the same function. Scanning the whole cache on every registration is
+ // quadratic, so stop once it outgrows that expectation. Larger caches are
+ // checked by AssumptionCacheTracker::verifyAnalysis() instead.
+ constexpr unsigned MaxAssumesToVerify = 64;
+ if (AssumeHandles.size() <= MaxAssumesToVerify) {
+ SmallPtrSet<const CallInst *, 16> Cached;
+ if (const char *Violation = findCacheViolation(F, AssumeHandles, Cached))
+ llvm_unreachable(Violation);
}
#endif
@@ -324,16 +349,18 @@ void AssumptionCacheTracker::verifyAnalysis() const {
if (!VerifyAssumptionCache)
return;
- SmallPtrSet<const CallInst *, 4> AssumptionSet;
for (const auto &I : AssumptionCaches) {
- for (auto &VH : I.second->assumptions())
- if (VH)
- AssumptionSet.insert(cast<CallInst>(VH));
+ const Function &F = cast<Function>(*I.first);
+
+ SmallPtrSet<const CallInst *, 4> Cached;
+ if (const char *Violation =
+ findCacheViolation(F, I.second->assumptions(), Cached))
+ report_fatal_error(Violation);
- for (const BasicBlock &B : cast<Function>(*I.first))
+ for (const BasicBlock &B : F)
for (const Instruction &II : B)
if (match(&II, m_Intrinsic<Intrinsic::assume>()) &&
- !AssumptionSet.count(cast<CallInst>(&II)))
+ !Cached.count(cast<CallInst>(&II)))
report_fatal_error("Assumption in scanned function not in cache");
}
}
diff --git a/llvm/lib/Transforms/Utils/CodeExtractor.cpp b/llvm/lib/Transforms/Utils/CodeExtractor.cpp
index db1676838854b..3e95c9baeeea4 100644
--- a/llvm/lib/Transforms/Utils/CodeExtractor.cpp
+++ b/llvm/lib/Transforms/Utils/CodeExtractor.cpp
@@ -2147,7 +2147,7 @@ bool CodeExtractor::verifyAssumptionCache(const Function &OldFunc,
// There shouldn't be any stale affected values in the assumption cache
// that were previously in the old function, but that have now been moved
// to the new function.
- for (auto AffectedValVH : AC->assumptionsFor(I->getOperand(0))) {
+ for (auto AffectedValVH : AC->allAssumptionsFor(I->getOperand(0))) {
auto *AffectedCI = dyn_cast_or_null<CallInst>(AffectedValVH);
if (!AffectedCI)
continue;
diff --git a/llvm/test/Analysis/AssumptionCache/max-assumes-per-value.ll b/llvm/test/Analysis/AssumptionCache/max-assumes-per-value.ll
new file mode 100644
index 0000000000000..773bd056c2db6
--- /dev/null
+++ b/llvm/test/Analysis/AssumptionCache/max-assumes-per-value.ll
@@ -0,0 +1,37 @@
+; RUN: opt < %s -S -passes=instsimplify -max-assumes-per-value=3 | FileCheck %s --check-prefixes=CHECK,USED
+; RUN: opt < %s -S -passes=instsimplify -max-assumes-per-value=2 | FileCheck %s --check-prefixes=CHECK,IGNORED
+
+; Analyses inspect every assumption returned for a value, so only the first
+; -max-assumes-per-value assumptions affecting it are used. Here the assumption
+; that proves the comparison is the third one.
+
+declare void @llvm.assume(i1)
+
+define i1 @assumes_within_limit(i32 %x) {
+; CHECK-LABEL: define i1 @assumes_within_limit(
+; USED: ret i1 true
+; IGNORED: ret i1 %cmp
+ %u1 = icmp ne i32 %x, 1234
+ call void @llvm.assume(i1 %u1)
+ %u2 = icmp ne i32 %x, 5678
+ call void @llvm.assume(i1 %u2)
+ %c = icmp sgt i32 %x, 41
+ call void @llvm.assume(i1 %c)
+ %cmp = icmp sgt i32 %x, 0
+ ret i1 %cmp
+}
+
+; The limit applies per value, so assumptions about %y are still used when %x is
+; affected by more of them than the limit allows.
+define i1 @limit_is_per_value(i32 %x, i32 %y) {
+; CHECK-LABEL: define i1 @limit_is_per_value(
+; CHECK: ret i1 true
+ %cx = icmp sgt i32 %x, 41
+ call void @llvm.assume(i1 %cx)
+ call void @llvm.assume(i1 %cx)
+ call void @llvm.assume(i1 %cx)
+ %cy = icmp sgt i32 %y, 41
+ call void @llvm.assume(i1 %cy)
+ %cmp = icmp sgt i32 %y, 0
+ ret i1 %cmp
+}
More information about the llvm-commits
mailing list