[llvm] [IVDescriptors] Implement MonotonicDescriptor (PR #214490)
Benjamin Maxwell via llvm-commits
llvm-commits at lists.llvm.org
Tue Aug 18 09:55:13 PDT 2026
https://github.com/MacDue updated https://github.com/llvm/llvm-project/pull/214490
>From 36c824cb380f7a05058f24f66642db7c394d2e01 Mon Sep 17 00:00:00 2001
From: Sergey Kachkov <sergey.kachkov at syntacore.com>
Date: Wed, 29 Jan 2025 15:27:01 +0300
Subject: [PATCH 01/15] [IVDescriptors] Implement MonotonicDescriptor
RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442
"Monotonic" variable is similar to induction variable, but its value is updated under some condition, e.g.:
```
int idx = 0;
for(int i = 0; i < n; ++i) {
// some uses of idx
if (cond)
++idx;
}
```
In this example, `i` is induction variable and `idx` is monotonic variable: it's updated only when cond == true. In LLVM IR, this looks like:
```
loop_header:
%monotonic_phi = [%start, %prehader], [ %chain_phi0, %latch]
step_bb:
%step = add/gep %monotonic_phi, %step_val
bbN:
%chain_phiN = [%step, %step_bb], [%monotonic_phi, %some_pred]
...
bb1:
%chain_phi1 = [%monotonic_phi, %some_pred], [%chain_phi2, %some_pred]
loop_latch:
%chain_phi0 = [%monotonic_phi, %some_pred], [%chain_phi1, %some_pred]
```
We start the analysis from the header phi (%monotonic_phi). Its backedge value (%chain_phi0) can be either unmodified (%monotonic_val) or the incremented one (%step) depending on some condition. Therefore, all incoming values in chain phi should be %monotonic_phi except the one that points to the next chain phi or the step instruction. The analysis stops when the correct step instruction is found (correct step instruction is in form add/gep %monotonic_phi, %step_val, and %step_val is loop-invariant). In this way, %monotonic_val is incremented on %step_val only when last chain phi "selects" %step incoming value; in other words, when step_bb -> bbN edge is executed. The condition of monotonic variable update is stored as such CFG edge (`getPredicateEdge()` method). The other MonotonicDescriptor methods are:
1. `getChain()` - returns set of chain phis (%chain_phi0, ..., %chain_phiN)
2. `getStepInst()` - returns %step instruction (either add or getlementptr)
3. `getExpr()` - returns SCEVAddRec in assumption that condition of update is always true (SCEVAddRec + condtion predicate fully describes evolution of monotonic variable in the loop)
MonotonicDescriptor also implements `isMonotonicVal` method. This routine will allow to recognize values which have monotonic phi as a transitive dependency, e.g. array subscript operators `arr[idx]`, where idx is monotonic phi. The descriptor of monotonic value is equivalent to the descriptor of monotonic phi, except its SCEV: we obtain the SCEV of monotonic value and replace the SCEVUnknown expression that corresponds to monotonic phi with the SCEVAddRec from monotonic phi descriptor. The resulted SCEV shows the evolution of monotonic value in loop iterations when predicate is true (similarly to SCEV).
*Restrictions*
1. We only support "post-increment" update of monotonic vars; in other words, uses of chain phis (other than in other chain phis) are prohibited. It's only alowed to use %monotonic_phi outside of described chain pattern.
2. We don't support `select` instructions in monotonic patterns (instead of chain phis). In theory this can be implemented, but on practice we want to use MonotonicDescriptor to recognize expandloads/compressstores, where step instruction is placed near memory instruction, so if-conversion to select is not applied there (because basic block can't be eliminated anyway). So, this restriction doesn't limit our abilities to recognize expandloads/compressstore patterns.
This is a continuation Sergey Kachkov's patch (#140720).
---
llvm/include/llvm/Analysis/IVDescriptors.h | 39 +++++
llvm/lib/Analysis/IVDescriptors.cpp | 121 ++++++++++++++
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 153 ++++++++++++++++++
3 files changed, 313 insertions(+)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index bad372421dfae..c644bac9a3ecc 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -29,6 +29,7 @@ class PredicatedScalarEvolution;
class ScalarEvolution;
class SCEV;
class SCEVPredicate;
+class SCEVAddRecExpr;
class StoreInst;
/// These are the kinds of recurrences that we support.
@@ -482,6 +483,44 @@ class InductionDescriptor {
SmallVector<const SCEVPredicate *, 2> NoWrapPredicates;
};
+/// A struct for saving information about monotonic variables.
+/// Monotonic variable can be considered as a "conditional" induction variable:
+/// its update happens only on loop iterations for which a certain predicate is
+/// satisfied. In this implementation the predicate is represented as an edge in
+/// loop CFG: variable is updated if this edge is executed on current loop
+/// iteration.
+class MonotonicDescriptor {
+public:
+ using Edge = std::pair<BasicBlock *, BasicBlock *>;
+
+ MonotonicDescriptor() = default;
+
+ const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
+ Instruction *getStepInst() const { return StepInst; }
+ Edge getPredicateEdge() const { return PredEdge; }
+ const SCEVAddRecExpr *getExpr() const { return Expr; }
+
+ /// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
+ /// is monotonic, the monotonic descriptor \p D will contain the data
+ /// describing this variable.
+ static bool isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+ /// Returns true if \p Val is a monotonic variable in the loop \p L (in this
+ /// case, the value should transitively contain monotonic phi as part of its
+ /// calculation).
+ static bool isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+private:
+ SmallPtrSet<PHINode *, 1> Chain;
+ Instruction *StepInst;
+ Edge PredEdge;
+ const SCEVAddRecExpr *Expr;
+
+ bool setSCEV(const SCEV *NewExpr);
+};
+
} // end namespace llvm
#endif // LLVM_ANALYSIS_IVDESCRIPTORS_H
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 800b64e9a29af..595e8b6dc1f78 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1696,3 +1696,124 @@ bool InductionDescriptor::isInductionPHI(
/*InductionBinOp=*/nullptr, /*Casts=*/nullptr, Preds);
return true;
}
+
+bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
+ auto *AddRec = dyn_cast<SCEVAddRecExpr>(NewExpr);
+ if (!AddRec || !AddRec->isAffine())
+ return false;
+ Expr = AddRec;
+ return true;
+}
+
+// Recognize monotonic phi variable by matching the following pattern:
+// loop_header:
+// %monotonic_phi = [%start, %preheader], [%chain_phi0, %latch]
+//
+// step_bb:
+// %step = add/gep %monotonic_phi, %step_val
+//
+// bbN:
+// %chain_phiN = [%monotonic_phi, ], [%step, ]
+//
+// ...
+//
+// bb1:
+// %chain_phi1 = [%monotonic_phi, ], [%chain_phi2, ]
+//
+// latch:
+// %chain_phi0 = [%monotonic_phi, %pred], [%chain_phi1, %pred]
+//
+// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
+// and predicate is CFG edge %step_bb -> %bbN.
+bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!PN->getType()->isIntOrPtrTy() || PN->getParent() != L->getHeader())
+ return false;
+ auto *BackEdgeInst =
+ dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
+ if (!BackEdgeInst)
+ return false;
+ PHINode *PHIChain = BackEdgeInst;
+ std::optional<std::pair<Edge, Value *>> Inc;
+ while (PHIChain) {
+ Desc.Chain.insert(PHIChain);
+ PHINode *NextPHIChain = nullptr;
+ for (auto [Block, Incoming] :
+ zip_equal(PHIChain->blocks(), PHIChain->incoming_values())) {
+ if (Incoming == PN)
+ continue;
+ if (!Incoming->hasOneUse())
+ return false;
+ if (auto *IncomingPHI = dyn_cast<PHINode>(Incoming)) {
+ if (NextPHIChain)
+ return false;
+ NextPHIChain = IncomingPHI;
+ continue;
+ }
+ if (Inc || NextPHIChain)
+ return false;
+ Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
+ }
+ PHIChain = NextPHIChain;
+ }
+ if (!Inc)
+ return false;
+ auto [PredEdge, StepOp] = *Inc;
+ auto *StepInst = dyn_cast<Instruction>(StepOp);
+ if (!StepInst)
+ return false;
+ Desc.StepInst = StepInst;
+ Desc.PredEdge = PredEdge;
+
+ // Construct SCEVAddRec for this value.
+ Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
+
+ Value *Step = nullptr;
+ bool StepMatch =
+ PN->getType()->isPointerTy()
+ ? match(StepInst, m_PtrAdd(m_Specific(PN), m_Value(Step)))
+ : match(StepInst, m_Add(m_Specific(PN), m_Value(Step)));
+ if (!StepMatch || !L->isLoopInvariant(Step))
+ return false;
+
+ SCEV::NoWrapFlags WrapFlags = SCEV::FlagAnyWrap;
+ if (auto *GEP = dyn_cast<GEPOperator>(StepInst)) {
+ if (GEP->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (GEP->hasNoUnsignedSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ } else if (auto *OBO = dyn_cast<OverflowingBinaryOperator>(StepInst)) {
+ if (OBO->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (OBO->hasNoSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ }
+
+ return Desc.setSCEV(
+ SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags));
+}
+
+bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!Val->getType()->isIntOrPtrTy() || L->isLoopInvariant(Val))
+ return false;
+ auto *CurInst = cast<Instruction>(Val);
+
+ auto LoopVariantVal = [&](Value *V, bool AllowRepeats) {
+ return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
+ };
+
+ while (!isa<PHINode>(CurInst)) {
+ CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
+ if (!CurInst)
+ return false;
+ };
+
+ if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
+ return false;
+
+ ValueToSCEVMapTy Map{{CurInst, Desc.getExpr()}};
+ return Desc.setSCEV(SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
+}
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index faf30fd322c10..753abb0c7b93d 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -10,6 +10,7 @@
#include "llvm/Analysis/AssumptionCache.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/AsmParser/Parser.h"
#include "llvm/IR/Dominators.h"
@@ -430,3 +431,155 @@ TEST(IVDescriptorsTest, InvariantStoreNoSCEV) {
EXPECT_FALSE(IsRdxPhi);
});
}
+
+TEST(IVDescriptorsTest, MonotonicIntVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %dst, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi i32 [ 0, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = add nsw i32 %monotonic, 1
+ %monotonic.prom = sext i32 %monotonic to i64
+ %arrayidx = getelementptr inbounds i32, ptr %dst, i64 %monotonic.prom
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi i32 [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ BBI = IfThen->begin();
+ Instruction *StepInst = &*BBI;
+ assert(StepInst->getName() == "inc");
+ Instruction *ExtInst = &*(++BBI);
+ assert(ExtInst->getName() == "monotonic.prom");
+ Instruction *GEPInst = &*(++BBI);
+ assert(GEPInst->getName() == "arrayidx");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ // Check %monotonic descriptor.
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &PhiChain = Desc.getChain();
+ EXPECT_TRUE(PhiChain.size() == 1 && PhiChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getConstant(Phi->getType(), 0);
+ auto *StepSCEV = SE.getConstant(Phi->getType(), 1);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ // Check %arrayidx descriptor.
+ bool IsMonotonicVal =
+ MonotonicDescriptor::isMonotonicVal(GEPInst, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicVal);
+ // Chain, StepInst and PredicateEdge are the same with %monotonic.
+ auto &ValChain = Desc.getChain();
+ EXPECT_TRUE(ValChain.size() == 1 && ValChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ StartSCEV = SE.getSCEV(F.getArg(0));
+ StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
+
+TEST(IVDescriptorsTest, MonotonicPtrVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %start, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi ptr [ %start, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = getelementptr inbounds i8, ptr %monotonic, i64 4
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi ptr [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ Instruction *StepInst = &*(IfThen->begin());
+ assert(StepInst->getName() == "inc");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &Chain = Desc.getChain();
+ EXPECT_TRUE(Chain.size() == 1 && Chain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getSCEV(F.getArg(0));
+ auto *StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
>From 4001eb04f6df134852e12b6cb45cd37639150c17 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Thu, 6 Aug 2026 19:31:23 +0000
Subject: [PATCH 02/15] remove ;
---
llvm/lib/Analysis/IVDescriptors.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 595e8b6dc1f78..db1ace2209b2c 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1809,7 +1809,7 @@ bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
if (!CurInst)
return false;
- };
+ }
if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
return false;
>From fa2878b9b8d81ee7fb785bde365292e91f7d2cd5 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Fri, 7 Aug 2026 09:17:28 +0000
Subject: [PATCH 03/15] Add docs
---
llvm/include/llvm/Analysis/IVDescriptors.h | 31 +++++++++++++++++++---
1 file changed, 27 insertions(+), 4 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index c644bac9a3ecc..4a1aa5830d51c 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -495,9 +495,19 @@ class MonotonicDescriptor {
MonotonicDescriptor() = default;
+ /// Returns the PHIs that feed into the backedge of the monotonic PHI.
const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
+
+ /// Returns the instruction that updates the value of the monotonic PHI.
Instruction *getStepInst() const { return StepInst; }
+
+ /// Returns the edge where the monotonic value/PHI is updated when taken.
Edge getPredicateEdge() const { return PredEdge; }
+
+ /// Returns the expression that represents the monotonic PHI (match with
+ /// isMonotonicPHI) or monotonic value (match with isMonotonicVal). Note: The
+ /// conditional update is represented with a plain SCEVAddRec within the
+ /// expression, this only holds along the predicated edge.
const SCEVAddRecExpr *getExpr() const { return Expr; }
/// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
@@ -507,17 +517,30 @@ class MonotonicDescriptor {
MonotonicDescriptor &Desc, ScalarEvolution &SE);
/// Returns true if \p Val is a monotonic variable in the loop \p L (in this
- /// case, the value should transitively contain monotonic phi as part of its
+ /// case, the value should transitively contain monotonic PHI as part of its
/// calculation).
static bool isMonotonicVal(Value *Val, const Loop *L,
MonotonicDescriptor &Desc, ScalarEvolution &SE);
private:
+ /// The PHIs that feed into the backedge update of the monotonic PHI.
SmallPtrSet<PHINode *, 1> Chain;
- Instruction *StepInst;
- Edge PredEdge;
- const SCEVAddRecExpr *Expr;
+ /// The instruction that updates the value of the monotonic PHI.
+ Instruction *StepInst = nullptr;
+
+ /// The predicated edge where the monotonic value/PHI is updated.
+ /// The common case is {StepInstBlock, StepInstBlock->getSingleSuccessor()}.
+ /// Note: StepInstBlock = StepInst->getParent().
+ Edge PredEdge = {};
+
+ /// Expression that represents the monotonic PHI (isMonotonicPHI) or monotonic
+ /// (isMonotonicVal). Within the expression, the conditional update is
+ /// represented as an (unconditional) SCEVAddRec.
+ const SCEVAddRecExpr *Expr = nullptr;
+
+ /// Set the SCEV expression for this descriptor. \p NewExpr must be an affine
+ /// SCEVAddRec.
bool setSCEV(const SCEV *NewExpr);
};
>From 999fc6b31638cfb103611a1bc7cab5d6dc64d235 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 10 Aug 2026 09:18:28 +0000
Subject: [PATCH 04/15] Fixups
---
llvm/include/llvm/Analysis/IVDescriptors.h | 9 +++++----
llvm/lib/Analysis/IVDescriptors.cpp | 3 ++-
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 4a1aa5830d51c..0d7e48e441f72 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -517,8 +517,8 @@ class MonotonicDescriptor {
MonotonicDescriptor &Desc, ScalarEvolution &SE);
/// Returns true if \p Val is a monotonic variable in the loop \p L (in this
- /// case, the value should transitively contain monotonic PHI as part of its
- /// calculation).
+ /// case, the value should transitively contain monotonic PHI as the only
+ /// loop-variant part/operand of its calculation).
static bool isMonotonicVal(Value *Val, const Loop *L,
MonotonicDescriptor &Desc, ScalarEvolution &SE);
@@ -539,8 +539,9 @@ class MonotonicDescriptor {
/// represented as an (unconditional) SCEVAddRec.
const SCEVAddRecExpr *Expr = nullptr;
- /// Set the SCEV expression for this descriptor. \p NewExpr must be an affine
- /// SCEVAddRec.
+ /// Set the SCEV expression for this descriptor. Returns true if the
+ /// expression was successfully updated, this requires \p NewExpr must be an
+ /// affine SCEVAddRec.
bool setSCEV(const SCEV *NewExpr);
};
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index db1ace2209b2c..ea7da3e89f5b4 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1751,6 +1751,7 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
NextPHIChain = IncomingPHI;
continue;
}
+ // Only one update/step is allowed. The unmodified value must be PN.
if (Inc || NextPHIChain)
return false;
Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
@@ -1801,7 +1802,7 @@ bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
return false;
auto *CurInst = cast<Instruction>(Val);
- auto LoopVariantVal = [&](Value *V, bool AllowRepeats) {
+ auto LoopVariantVal = [&](Value *V, bool /*AllowRepeats*/) {
return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
};
>From 188ed064e289272e867f71a366aecd959d21a545 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Tue, 11 Aug 2026 12:16:42 +0000
Subject: [PATCH 05/15] Fixups
---
llvm/include/llvm/Analysis/IVDescriptors.h | 6 +++
llvm/lib/Analysis/IVDescriptors.cpp | 45 ++++++++++++++--------
2 files changed, 35 insertions(+), 16 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 0d7e48e441f72..081a4412550da 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -495,6 +495,12 @@ class MonotonicDescriptor {
MonotonicDescriptor() = default;
+ MonotonicDescriptor(SmallPtrSetImpl<PHINode *> &Chain, Instruction *StepInst,
+ Edge PredEdge, const SCEV *Expr)
+ : Chain(llvm::from_range, Chain), StepInst(StepInst), PredEdge(PredEdge) {
+ setSCEV(Expr);
+ }
+
/// Returns the PHIs that feed into the backedge of the monotonic PHI.
const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index ea7da3e89f5b4..8e49666bf7415 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1707,21 +1707,30 @@ bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
// Recognize monotonic phi variable by matching the following pattern:
// loop_header:
-// %monotonic_phi = [%start, %preheader], [%chain_phi0, %latch]
+// %monotonic_phi = phi [ %start, %preheader ], [ %chain_phi0, %latch ]
+// br i1 %do_step, label %step_bb, label %bbN
//
// step_bb:
// %step = add/gep %monotonic_phi, %step_val
+// br label %bbN
//
// bbN:
-// %chain_phiN = [%monotonic_phi, ], [%step, ]
+// %chain_phiN = phi [ %monotonic_phi, %loop_header ], [ %step, %step_bb ]
+// br label %bb(N-1)
//
// ...
//
+// bb2:
+// %chain_phi2 = phi [ %monotonic_phi, %pred2 ], [ %chain_phi3, %bb3 ]
+// br label %bb1
+//
// bb1:
-// %chain_phi1 = [%monotonic_phi, ], [%chain_phi2, ]
+// %chain_phi1 = phi [ %monotonic_phi, %pred1 ], [ %chain_phi2, %bb2 ]
+// br label %latch
//
// latch:
-// %chain_phi0 = [%monotonic_phi, %pred], [%chain_phi1, %pred]
+// %chain_phi0 = phi [ %monotonic_phi, %pred0 ], [ %chain_phi1, %bb1 ]
+// br label %loop_header
//
// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
// and predicate is CFG edge %step_bb -> %bbN.
@@ -1734,10 +1743,14 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
if (!BackEdgeInst)
return false;
+
+ Edge PredEdge;
+ Value *StepOp = nullptr;
+ SmallPtrSet<PHINode *, 1> Chain;
PHINode *PHIChain = BackEdgeInst;
- std::optional<std::pair<Edge, Value *>> Inc;
+
while (PHIChain) {
- Desc.Chain.insert(PHIChain);
+ Chain.insert(PHIChain);
PHINode *NextPHIChain = nullptr;
for (auto [Block, Incoming] :
zip_equal(PHIChain->blocks(), PHIChain->incoming_values())) {
@@ -1752,20 +1765,17 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
continue;
}
// Only one update/step is allowed. The unmodified value must be PN.
- if (Inc || NextPHIChain)
+ if (StepOp || NextPHIChain)
return false;
- Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
+ PredEdge = Edge{Block, PHIChain->getParent()};
+ StepOp = Incoming;
}
PHIChain = NextPHIChain;
}
- if (!Inc)
- return false;
- auto [PredEdge, StepOp] = *Inc;
- auto *StepInst = dyn_cast<Instruction>(StepOp);
+
+ auto *StepInst = dyn_cast_if_present<Instruction>(StepOp);
if (!StepInst)
return false;
- Desc.StepInst = StepInst;
- Desc.PredEdge = PredEdge;
// Construct SCEVAddRec for this value.
Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
@@ -1791,8 +1801,11 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
}
- return Desc.setSCEV(
- SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags));
+ const SCEV *PhiSCEV =
+ SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags);
+
+ Desc = MonotonicDescriptor(Chain, StepInst, PredEdge, PhiSCEV);
+ return Desc.getExpr() != nullptr;
}
bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
>From d57ab4a9522e6a0cb57ab40683c69e6dbb3ed40a Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Fri, 14 Aug 2026 11:04:56 +0000
Subject: [PATCH 06/15] Fixups
---
llvm/include/llvm/Analysis/IVDescriptors.h | 4 ++--
llvm/lib/Analysis/IVDescriptors.cpp | 2 +-
2 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 081a4412550da..953b88e39bb72 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -498,7 +498,7 @@ class MonotonicDescriptor {
MonotonicDescriptor(SmallPtrSetImpl<PHINode *> &Chain, Instruction *StepInst,
Edge PredEdge, const SCEV *Expr)
: Chain(llvm::from_range, Chain), StepInst(StepInst), PredEdge(PredEdge) {
- setSCEV(Expr);
+ setIfAffineAddRec(Expr);
}
/// Returns the PHIs that feed into the backedge of the monotonic PHI.
@@ -548,7 +548,7 @@ class MonotonicDescriptor {
/// Set the SCEV expression for this descriptor. Returns true if the
/// expression was successfully updated, this requires \p NewExpr must be an
/// affine SCEVAddRec.
- bool setSCEV(const SCEV *NewExpr);
+ bool setIfAffineAddRec(const SCEV *NewExpr);
};
} // end namespace llvm
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 8e49666bf7415..bd1d70b4e3fdf 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1697,7 +1697,7 @@ bool InductionDescriptor::isInductionPHI(
return true;
}
-bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
+bool MonotonicDescriptor::setIfAffineAddRec(const SCEV *NewExpr) {
auto *AddRec = dyn_cast<SCEVAddRecExpr>(NewExpr);
if (!AddRec || !AddRec->isAffine())
return false;
>From 7a4a3a94ac6b38d69fe211c09693fd12de3122f3 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Fri, 14 Aug 2026 14:33:25 +0000
Subject: [PATCH 07/15] Fixups
---
llvm/lib/Analysis/IVDescriptors.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index bd1d70b4e3fdf..d824440a7c2cb 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1829,5 +1829,6 @@ bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
return false;
ValueToSCEVMapTy Map{{CurInst, Desc.getExpr()}};
- return Desc.setSCEV(SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
+ return Desc.setIfAffineAddRec(
+ SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
}
>From 2a520ba138d737f23cd88020ced5184594fd927f Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 17 Aug 2026 10:18:40 +0000
Subject: [PATCH 08/15] Rework desc to check users
---
llvm/include/llvm/Analysis/IVDescriptors.h | 55 +++---
llvm/lib/Analysis/IVDescriptors.cpp | 167 +++++++++++++-----
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 17 +-
3 files changed, 166 insertions(+), 73 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 953b88e39bb72..aad9f6ae21f10 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -495,23 +495,30 @@ class MonotonicDescriptor {
MonotonicDescriptor() = default;
- MonotonicDescriptor(SmallPtrSetImpl<PHINode *> &Chain, Instruction *StepInst,
- Edge PredEdge, const SCEV *Expr)
- : Chain(llvm::from_range, Chain), StepInst(StepInst), PredEdge(PredEdge) {
- setIfAffineAddRec(Expr);
- }
+ MonotonicDescriptor(
+ const SmallPtrSetImpl<PHINode *> &Chain,
+ const DenseMap<Instruction *, const SCEV *> &CompressedMemOps,
+ Instruction *StepInst, Edge PredEdge, const SCEVAddRecExpr *Expr)
+ : Chain(llvm::from_range, Chain), CompressedMemOps(CompressedMemOps),
+ StepInst(StepInst), PredEdge(PredEdge), Expr(Expr) {}
/// Returns the PHIs that feed into the backedge of the monotonic PHI.
const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
+ // Returns memory operations whose addresses are derived from this monotonic
+ // PHI. The keys are load or store instructions, the values are SCEVAddRecs
+ // that represent the pointer operand along the predicated edge.
+ const DenseMap<Instruction *, const SCEV *> &getCompressedMemoryOps() const {
+ return CompressedMemOps;
+ }
+
/// Returns the instruction that updates the value of the monotonic PHI.
Instruction *getStepInst() const { return StepInst; }
/// Returns the edge where the monotonic value/PHI is updated when taken.
Edge getPredicateEdge() const { return PredEdge; }
- /// Returns the expression that represents the monotonic PHI (match with
- /// isMonotonicPHI) or monotonic value (match with isMonotonicVal). Note: The
+ /// Returns the expression that represents the monotonic PHI. Note: The
/// conditional update is represented with a plain SCEVAddRec within the
/// expression, this only holds along the predicated edge.
const SCEVAddRecExpr *getExpr() const { return Expr; }
@@ -522,16 +529,13 @@ class MonotonicDescriptor {
static bool isMonotonicPHI(PHINode *PN, const Loop *L,
MonotonicDescriptor &Desc, ScalarEvolution &SE);
- /// Returns true if \p Val is a monotonic variable in the loop \p L (in this
- /// case, the value should transitively contain monotonic PHI as the only
- /// loop-variant part/operand of its calculation).
- static bool isMonotonicVal(Value *Val, const Loop *L,
- MonotonicDescriptor &Desc, ScalarEvolution &SE);
-
private:
/// The PHIs that feed into the backedge update of the monotonic PHI.
SmallPtrSet<PHINode *, 1> Chain;
+ /// Memory operations whose addresses are derived from this monotonic PHI.
+ DenseMap<Instruction *, const SCEV *> CompressedMemOps;
+
/// The instruction that updates the value of the monotonic PHI.
Instruction *StepInst = nullptr;
@@ -540,15 +544,26 @@ class MonotonicDescriptor {
/// Note: StepInstBlock = StepInst->getParent().
Edge PredEdge = {};
- /// Expression that represents the monotonic PHI (isMonotonicPHI) or monotonic
- /// (isMonotonicVal). Within the expression, the conditional update is
- /// represented as an (unconditional) SCEVAddRec.
+ /// Expression that represents the monotonic PHI. Within the expression, the
+ /// conditional update is represented as an (unconditional) SCEVAddRec.
const SCEVAddRecExpr *Expr = nullptr;
- /// Set the SCEV expression for this descriptor. Returns true if the
- /// expression was successfully updated, this requires \p NewExpr must be an
- /// affine SCEVAddRec.
- bool setIfAffineAddRec(const SCEV *NewExpr);
+ /// Verifies \p PN is a monotonic PHI and collects the PHIs within the chain.
+ /// Returns the StepInst, PHI SCEV expression, and the predicated edge.
+ static std::pair<Instruction *, const SCEV *>
+ CollectMonotonicPHIChain(PHINode *PN, const Loop *L, PHINode *BackEdgeInst,
+ SmallPtrSetImpl<PHINode *> &Chain, Edge &PredEdge,
+ ScalarEvolution &SE);
+
+ /// Collects the memory operations whose addresses are derived from \p PN.
+ /// The memory operations and SCEV expressions for their pointer operands are
+ /// placed in \p CompressedMemOps. Returns true if no unexpected users of \p
+ /// PN were found.
+ static bool CollectCompressedMemOpUsers(
+ PHINode *PN, const Loop *L, Edge PredEdge,
+ const SmallPtrSetImpl<PHINode *> &Chain, const SCEV *PhiSCEV,
+ ScalarEvolution &SE,
+ DenseMap<Instruction *, const SCEV *> &CompressedMemOps);
};
} // end namespace llvm
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index d824440a7c2cb..24a469daac439 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1697,12 +1697,12 @@ bool InductionDescriptor::isInductionPHI(
return true;
}
-bool MonotonicDescriptor::setIfAffineAddRec(const SCEV *NewExpr) {
- auto *AddRec = dyn_cast<SCEVAddRecExpr>(NewExpr);
- if (!AddRec || !AddRec->isAffine())
- return false;
- Expr = AddRec;
- return true;
+static bool hasUniqueLoopVariantOperand(Value *Cur, Instruction *I,
+ const Loop *L) {
+ auto LoopVariantOp = [&](Value *V, bool /*AllowRepeats*/) -> Value * {
+ return L->isLoopInvariant(V) ? nullptr : V;
+ };
+ return find_singleton<Value>(I->operands(), LoopVariantOp) == Cur;
}
// Recognize monotonic phi variable by matching the following pattern:
@@ -1732,21 +1732,15 @@ bool MonotonicDescriptor::setIfAffineAddRec(const SCEV *NewExpr) {
// %chain_phi0 = phi [ %monotonic_phi, %pred0 ], [ %chain_phi1, %bb1 ]
// br label %loop_header
//
-// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
-// and predicate is CFG edge %step_bb -> %bbN.
-bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
- MonotonicDescriptor &Desc,
- ScalarEvolution &SE) {
- if (!PN->getType()->isIntOrPtrTy() || PN->getParent() != L->getHeader())
- return false;
- auto *BackEdgeInst =
- dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
- if (!BackEdgeInst)
- return false;
-
- Edge PredEdge;
+// For this pattern, monotonic phi is described by {%start, +, %step}
+// recurrence and predicate is CFG edge %step_bb -> %bbN.
+std::pair<Instruction *, const SCEV *>
+MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
+ PHINode *BackEdgeInst,
+ SmallPtrSetImpl<PHINode *> &Chain,
+ Edge &PredEdge,
+ ScalarEvolution &SE) {
Value *StepOp = nullptr;
- SmallPtrSet<PHINode *, 1> Chain;
PHINode *PHIChain = BackEdgeInst;
while (PHIChain) {
@@ -1757,16 +1751,16 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
if (Incoming == PN)
continue;
if (!Incoming->hasOneUse())
- return false;
+ return {};
if (auto *IncomingPHI = dyn_cast<PHINode>(Incoming)) {
if (NextPHIChain)
- return false;
+ return {};
NextPHIChain = IncomingPHI;
continue;
}
// Only one update/step is allowed. The unmodified value must be PN.
if (StepOp || NextPHIChain)
- return false;
+ return {};
PredEdge = Edge{Block, PHIChain->getParent()};
StepOp = Incoming;
}
@@ -1775,7 +1769,7 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
auto *StepInst = dyn_cast_if_present<Instruction>(StepOp);
if (!StepInst)
- return false;
+ return {};
// Construct SCEVAddRec for this value.
Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
@@ -1786,7 +1780,7 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
? match(StepInst, m_PtrAdd(m_Specific(PN), m_Value(Step)))
: match(StepInst, m_Add(m_Specific(PN), m_Value(Step)));
if (!StepMatch || !L->isLoopInvariant(Step))
- return false;
+ return {};
SCEV::NoWrapFlags WrapFlags = SCEV::FlagAnyWrap;
if (auto *GEP = dyn_cast<GEPOperator>(StepInst)) {
@@ -1803,32 +1797,121 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
const SCEV *PhiSCEV =
SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags);
+ return {StepInst, PhiSCEV};
+}
+
+bool MonotonicDescriptor::CollectCompressedMemOpUsers(
+ PHINode *PN, const Loop *L, Edge PredEdge,
+ const SmallPtrSetImpl<PHINode *> &Chain, const SCEV *PhiSCEV,
+ ScalarEvolution &SE,
+ DenseMap<Instruction *, const SCEV *> &CompressedMemOps) {
+ ValueToSCEVMapTy PhiMap{{PN, PhiSCEV}};
+
+ auto GetCompressedPtrSCEV = [&](Instruction *MemI) -> const SCEV * {
+ // Check that the memory operation has the same predicate as the step.
+ // TODO: Relax these restrictions.
+ BasicBlock *AccessBB = MemI->getParent();
+ if (PredEdge != Edge(AccessBB, AccessBB->getUniqueSuccessor()))
+ return nullptr;
+
+ const SCEV *PtrSCEV = SCEVParameterRewriter::rewrite(
+ SE.getSCEV(getLoadStorePointerOperand(MemI)), SE, PhiMap);
+ auto *AddRec = dyn_cast<SCEVAddRecExpr>(PtrSCEV);
+ if (!AddRec || !AddRec->isAffine())
+ return nullptr;
+
+ // Check if pointer step equals access size.
+ SCEVUse Step = AddRec->getStepRecurrence(SE);
+
+ if (Step != SE.getSizeOfExpr(Step->getType(), getLoadStoreType(MemI)))
+ return nullptr;
+
+ return PtrSCEV;
+ };
+
+ SmallPtrSet<Use *, 16> Seen;
+ SmallVector<Use *> Worklist{make_pointer_range(PN->uses())};
+
+ while (!Worklist.empty()) {
+ Use *U = Worklist.pop_back_val();
+ if (!Seen.insert(U).second)
+ continue;
+
+ auto *I = dyn_cast<Instruction>(U->getUser());
+ if (!I)
+ continue;
+
+ if (isa<LoadInst, StoreInst>(I)) {
+ // Disallow any store using the PN as the stored value.
+ if (auto *SI = dyn_cast<StoreInst>(I);
+ SI && SI->getValueOperand() == U->get())
+ return false;
+
+ const SCEV *CompressedPtr = GetCompressedPtrSCEV(I);
+ if (!CompressedPtr)
+ return false;
+ CompressedMemOps.insert({I, CompressedPtr});
+ continue;
+ }
- Desc = MonotonicDescriptor(Chain, StepInst, PredEdge, PhiSCEV);
- return Desc.getExpr() != nullptr;
+ // Allow any phi that's part of the phi chain. Ignore it's users as they
+ // should only be other operations in the chain.
+ if (auto *Phi = dyn_cast<PHINode>(I)) {
+ if (Chain.contains(Phi))
+ continue;
+ return false;
+ }
+
+ // Non-memory users may use any opcode (select/and/or/etc.), but they must
+ // propagate Cur as their only loop-varying input. That prevents mixing in a
+ // second loop-varying term; GetCompressedPtrSCEV then rewrites the full
+ // leaf pointer SCEV and rejects it unless the entire address still
+ // simplifies to the required affine AddRec.
+ if (I->use_empty() || !hasUniqueLoopVariantOperand(U->get(), I, L))
+ return false;
+ append_range(Worklist, make_pointer_range(I->uses()));
+ }
+
+ return true;
}
-bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
+bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
MonotonicDescriptor &Desc,
ScalarEvolution &SE) {
- if (!Val->getType()->isIntOrPtrTy() || L->isLoopInvariant(Val))
+ if (!PN->getType()->isIntOrPtrTy() || PN->getParent() != L->getHeader())
+ return false;
+ auto *BackEdgeInst =
+ dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
+ if (!BackEdgeInst)
return false;
- auto *CurInst = cast<Instruction>(Val);
-
- auto LoopVariantVal = [&](Value *V, bool /*AllowRepeats*/) {
- return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
- };
- while (!isa<PHINode>(CurInst)) {
- CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
- if (!CurInst)
- return false;
+ for (User *U : BackEdgeInst->users()) {
+ auto *UI = dyn_cast<Instruction>(U);
+ if (UI == PN || (UI && !L->contains(UI)))
+ continue;
+ return false;
}
- if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
+ // Only allow the monotonic recurrence to feed the PHI chain or memory
+ // operations addressed by a compressed pointer. Save those memory operations
+ // on the descriptor.
+ Edge PredEdge;
+ SmallPtrSet<PHINode *, 1> Chain;
+ auto [StepInst, PhiSCEV] =
+ CollectMonotonicPHIChain(PN, L, BackEdgeInst, Chain, PredEdge, SE);
+ if (!StepInst)
return false;
- ValueToSCEVMapTy Map{{CurInst, Desc.getExpr()}};
- return Desc.setIfAffineAddRec(
- SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
+ const SCEVAddRecExpr *PhiAddRec = dyn_cast<SCEVAddRecExpr>(PhiSCEV);
+ if (!PhiAddRec || !PhiAddRec->isAffine())
+ return false;
+
+ DenseMap<Instruction *, const SCEV *> CompressedMemOps;
+ if (!CollectCompressedMemOpUsers(PN, L, PredEdge, Chain, PhiAddRec, SE,
+ CompressedMemOps))
+ return false;
+
+ Desc = MonotonicDescriptor(Chain, CompressedMemOps, StepInst, PredEdge,
+ PhiAddRec);
+ return true;
}
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index 753abb0c7b93d..a1b9928271f83 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -451,6 +451,7 @@ if.then:
%inc = add nsw i32 %monotonic, 1
%monotonic.prom = sext i32 %monotonic to i64
%arrayidx = getelementptr inbounds i32, ptr %dst, i64 %monotonic.prom
+ store i32 10, ptr %arrayidx, align 4
br label %for.inc
for.inc:
@@ -502,19 +503,13 @@ for.end:
auto *StepSCEV = SE.getConstant(Phi->getType(), 1);
EXPECT_EQ(Desc.getExpr(),
SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
- // Check %arrayidx descriptor.
- bool IsMonotonicVal =
- MonotonicDescriptor::isMonotonicVal(GEPInst, L, Desc, SE);
- EXPECT_TRUE(IsMonotonicVal);
- // Chain, StepInst and PredicateEdge are the same with %monotonic.
- auto &ValChain = Desc.getChain();
- EXPECT_TRUE(ValChain.size() == 1 && ValChain.contains(ChainPhi));
- EXPECT_EQ(Desc.getStepInst(), StepInst);
- EXPECT_EQ(Desc.getPredicateEdge(),
- MonotonicDescriptor::Edge(IfThen, IfEnd));
+ // Check %arrayidx use.
+ EXPECT_EQ(Desc.getCompressedMemoryOps().size(), 1U);
+ auto [StoreInst, PtrSCEV] = *Desc.getCompressedMemoryOps().begin();
+ EXPECT_EQ(getPointerOperand(StoreInst), GEPInst);
StartSCEV = SE.getSCEV(F.getArg(0));
StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
- EXPECT_EQ(Desc.getExpr(),
+ EXPECT_EQ(PtrSCEV,
SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
});
}
>From 6586ae9e8d70ec9aaf10cd089b04aef48394a63a Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 17 Aug 2026 20:33:34 +0000
Subject: [PATCH 09/15] Fixups
---
llvm/lib/Analysis/IVDescriptors.cpp | 20 ++++++++------------
1 file changed, 8 insertions(+), 12 deletions(-)
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 24a469daac439..ebc06aeb6d0bd 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1697,14 +1697,6 @@ bool InductionDescriptor::isInductionPHI(
return true;
}
-static bool hasUniqueLoopVariantOperand(Value *Cur, Instruction *I,
- const Loop *L) {
- auto LoopVariantOp = [&](Value *V, bool /*AllowRepeats*/) -> Value * {
- return L->isLoopInvariant(V) ? nullptr : V;
- };
- return find_singleton<Value>(I->operands(), LoopVariantOp) == Cur;
-}
-
// Recognize monotonic phi variable by matching the following pattern:
// loop_header:
// %monotonic_phi = phi [ %start, %preheader ], [ %chain_phi0, %latch ]
@@ -1837,9 +1829,8 @@ bool MonotonicDescriptor::CollectCompressedMemOpUsers(
if (!Seen.insert(U).second)
continue;
- auto *I = dyn_cast<Instruction>(U->getUser());
- if (!I)
- continue;
+ Value *CurrentVal = U->get();
+ auto *I = cast<Instruction>(U->getUser());
if (isa<LoadInst, StoreInst>(I)) {
// Disallow any store using the PN as the stored value.
@@ -1862,12 +1853,17 @@ bool MonotonicDescriptor::CollectCompressedMemOpUsers(
return false;
}
+ auto LoopVariantOp = [&](Value *V, bool /*AllowRepeats*/) -> Value * {
+ return L->isLoopInvariant(V) ? nullptr : V;
+ };
+
// Non-memory users may use any opcode (select/and/or/etc.), but they must
// propagate Cur as their only loop-varying input. That prevents mixing in a
// second loop-varying term; GetCompressedPtrSCEV then rewrites the full
// leaf pointer SCEV and rejects it unless the entire address still
// simplifies to the required affine AddRec.
- if (I->use_empty() || !hasUniqueLoopVariantOperand(U->get(), I, L))
+ if (I->use_empty() ||
+ find_singleton<Value>(I->operands(), LoopVariantOp) != CurrentVal)
return false;
append_range(Worklist, make_pointer_range(I->uses()));
}
>From e2156fdc953e2c3b8dab0a3df5f8305d2565e6a3 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 17 Aug 2026 20:38:32 +0000
Subject: [PATCH 10/15] Fixups
---
llvm/lib/Analysis/IVDescriptors.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index ebc06aeb6d0bd..2339eb731591d 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1858,9 +1858,9 @@ bool MonotonicDescriptor::CollectCompressedMemOpUsers(
};
// Non-memory users may use any opcode (select/and/or/etc.), but they must
- // propagate Cur as their only loop-varying input. That prevents mixing in a
- // second loop-varying term; GetCompressedPtrSCEV then rewrites the full
- // leaf pointer SCEV and rejects it unless the entire address still
+ // only have CurrentVal as their only loop-varying input. That prevents
+ // mixing in a second loop-varying term. GetCompressedPtrSCEV rewrites the
+ // full leaf pointer SCEV and rejects it unless the entire address still
// simplifies to the required affine AddRec.
if (I->use_empty() ||
find_singleton<Value>(I->operands(), LoopVariantOp) != CurrentVal)
>From 456eb5656a11c561f0b7135fe447fa8e6e40c36b Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 17 Aug 2026 20:40:26 +0000
Subject: [PATCH 11/15] Fixups
---
llvm/lib/Analysis/IVDescriptors.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 2339eb731591d..e38b002132566 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1835,7 +1835,7 @@ bool MonotonicDescriptor::CollectCompressedMemOpUsers(
if (isa<LoadInst, StoreInst>(I)) {
// Disallow any store using the PN as the stored value.
if (auto *SI = dyn_cast<StoreInst>(I);
- SI && SI->getValueOperand() == U->get())
+ SI && SI->getValueOperand() == CurrentVal)
return false;
const SCEV *CompressedPtr = GetCompressedPtrSCEV(I);
>From 72025643af13038b4e229b1a36ed5a11821515ce Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Tue, 18 Aug 2026 12:56:29 +0000
Subject: [PATCH 12/15] Rework
---
llvm/include/llvm/Analysis/IVDescriptors.h | 33 ++++++-------------
llvm/lib/Analysis/IVDescriptors.cpp | 23 +++----------
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 4 ---
3 files changed, 15 insertions(+), 45 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index aad9f6ae21f10..a9995027c14fe 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -486,28 +486,24 @@ class InductionDescriptor {
/// A struct for saving information about monotonic variables.
/// Monotonic variable can be considered as a "conditional" induction variable:
/// its update happens only on loop iterations for which a certain predicate is
-/// satisfied. In this implementation the predicate is represented as an edge in
-/// loop CFG: variable is updated if this edge is executed on current loop
-/// iteration.
+/// satisfied.
class MonotonicDescriptor {
public:
- using Edge = std::pair<BasicBlock *, BasicBlock *>;
-
MonotonicDescriptor() = default;
MonotonicDescriptor(
const SmallPtrSetImpl<PHINode *> &Chain,
const DenseMap<Instruction *, const SCEV *> &CompressedMemOps,
- Instruction *StepInst, Edge PredEdge, const SCEVAddRecExpr *Expr)
+ Instruction *StepInst, const SCEVAddRecExpr *Expr)
: Chain(llvm::from_range, Chain), CompressedMemOps(CompressedMemOps),
- StepInst(StepInst), PredEdge(PredEdge), Expr(Expr) {}
+ StepInst(StepInst), Expr(Expr) {}
/// Returns the PHIs that feed into the backedge of the monotonic PHI.
const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
// Returns memory operations whose addresses are derived from this monotonic
// PHI. The keys are load or store instructions, the values are SCEVAddRecs
- // that represent the pointer operand along the predicated edge.
+ // that represent how the pointer is updated by StepInst.
const DenseMap<Instruction *, const SCEV *> &getCompressedMemoryOps() const {
return CompressedMemOps;
}
@@ -515,12 +511,9 @@ class MonotonicDescriptor {
/// Returns the instruction that updates the value of the monotonic PHI.
Instruction *getStepInst() const { return StepInst; }
- /// Returns the edge where the monotonic value/PHI is updated when taken.
- Edge getPredicateEdge() const { return PredEdge; }
-
/// Returns the expression that represents the monotonic PHI. Note: The
- /// conditional update is represented with a plain SCEVAddRec within the
- /// expression, this only holds along the predicated edge.
+ /// conditional update is represented with a plain SCEVAddRec. This only holds
+ /// on iterations where the monotonic is updated by StepInst.
const SCEVAddRecExpr *getExpr() const { return Expr; }
/// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
@@ -539,20 +532,15 @@ class MonotonicDescriptor {
/// The instruction that updates the value of the monotonic PHI.
Instruction *StepInst = nullptr;
- /// The predicated edge where the monotonic value/PHI is updated.
- /// The common case is {StepInstBlock, StepInstBlock->getSingleSuccessor()}.
- /// Note: StepInstBlock = StepInst->getParent().
- Edge PredEdge = {};
-
/// Expression that represents the monotonic PHI. Within the expression, the
/// conditional update is represented as an (unconditional) SCEVAddRec.
const SCEVAddRecExpr *Expr = nullptr;
/// Verifies \p PN is a monotonic PHI and collects the PHIs within the chain.
- /// Returns the StepInst, PHI SCEV expression, and the predicated edge.
+ /// Returns the StepInst and a SCEV expression for \p PN.
static std::pair<Instruction *, const SCEV *>
CollectMonotonicPHIChain(PHINode *PN, const Loop *L, PHINode *BackEdgeInst,
- SmallPtrSetImpl<PHINode *> &Chain, Edge &PredEdge,
+ SmallPtrSetImpl<PHINode *> &Chain,
ScalarEvolution &SE);
/// Collects the memory operations whose addresses are derived from \p PN.
@@ -560,9 +548,8 @@ class MonotonicDescriptor {
/// placed in \p CompressedMemOps. Returns true if no unexpected users of \p
/// PN were found.
static bool CollectCompressedMemOpUsers(
- PHINode *PN, const Loop *L, Edge PredEdge,
- const SmallPtrSetImpl<PHINode *> &Chain, const SCEV *PhiSCEV,
- ScalarEvolution &SE,
+ PHINode *PN, const Loop *L, const SmallPtrSetImpl<PHINode *> &Chain,
+ const SCEV *PhiSCEV, ScalarEvolution &SE,
DenseMap<Instruction *, const SCEV *> &CompressedMemOps);
};
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index e38b002132566..79c5feb6d5b22 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1730,7 +1730,6 @@ std::pair<Instruction *, const SCEV *>
MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
PHINode *BackEdgeInst,
SmallPtrSetImpl<PHINode *> &Chain,
- Edge &PredEdge,
ScalarEvolution &SE) {
Value *StepOp = nullptr;
PHINode *PHIChain = BackEdgeInst;
@@ -1753,7 +1752,6 @@ MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
// Only one update/step is allowed. The unmodified value must be PN.
if (StepOp || NextPHIChain)
return {};
- PredEdge = Edge{Block, PHIChain->getParent()};
StepOp = Incoming;
}
PHIChain = NextPHIChain;
@@ -1763,7 +1761,6 @@ MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
if (!StepInst)
return {};
- // Construct SCEVAddRec for this value.
Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
Value *Step = nullptr;
@@ -1793,19 +1790,12 @@ MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
}
bool MonotonicDescriptor::CollectCompressedMemOpUsers(
- PHINode *PN, const Loop *L, Edge PredEdge,
- const SmallPtrSetImpl<PHINode *> &Chain, const SCEV *PhiSCEV,
- ScalarEvolution &SE,
+ PHINode *PN, const Loop *L, const SmallPtrSetImpl<PHINode *> &Chain,
+ const SCEV *PhiSCEV, ScalarEvolution &SE,
DenseMap<Instruction *, const SCEV *> &CompressedMemOps) {
ValueToSCEVMapTy PhiMap{{PN, PhiSCEV}};
auto GetCompressedPtrSCEV = [&](Instruction *MemI) -> const SCEV * {
- // Check that the memory operation has the same predicate as the step.
- // TODO: Relax these restrictions.
- BasicBlock *AccessBB = MemI->getParent();
- if (PredEdge != Edge(AccessBB, AccessBB->getUniqueSuccessor()))
- return nullptr;
-
const SCEV *PtrSCEV = SCEVParameterRewriter::rewrite(
SE.getSCEV(getLoadStorePointerOperand(MemI)), SE, PhiMap);
auto *AddRec = dyn_cast<SCEVAddRecExpr>(PtrSCEV);
@@ -1814,7 +1804,6 @@ bool MonotonicDescriptor::CollectCompressedMemOpUsers(
// Check if pointer step equals access size.
SCEVUse Step = AddRec->getStepRecurrence(SE);
-
if (Step != SE.getSizeOfExpr(Step->getType(), getLoadStoreType(MemI)))
return nullptr;
@@ -1891,10 +1880,9 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
// Only allow the monotonic recurrence to feed the PHI chain or memory
// operations addressed by a compressed pointer. Save those memory operations
// on the descriptor.
- Edge PredEdge;
SmallPtrSet<PHINode *, 1> Chain;
auto [StepInst, PhiSCEV] =
- CollectMonotonicPHIChain(PN, L, BackEdgeInst, Chain, PredEdge, SE);
+ CollectMonotonicPHIChain(PN, L, BackEdgeInst, Chain, SE);
if (!StepInst)
return false;
@@ -1903,11 +1891,10 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
return false;
DenseMap<Instruction *, const SCEV *> CompressedMemOps;
- if (!CollectCompressedMemOpUsers(PN, L, PredEdge, Chain, PhiAddRec, SE,
+ if (!CollectCompressedMemOpUsers(PN, L, Chain, PhiAddRec, SE,
CompressedMemOps))
return false;
- Desc = MonotonicDescriptor(Chain, CompressedMemOps, StepInst, PredEdge,
- PhiAddRec);
+ Desc = MonotonicDescriptor(Chain, CompressedMemOps, StepInst, PhiAddRec);
return true;
}
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index a1b9928271f83..28c563830e172 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -497,8 +497,6 @@ for.end:
auto &PhiChain = Desc.getChain();
EXPECT_TRUE(PhiChain.size() == 1 && PhiChain.contains(ChainPhi));
EXPECT_EQ(Desc.getStepInst(), StepInst);
- EXPECT_EQ(Desc.getPredicateEdge(),
- MonotonicDescriptor::Edge(IfThen, IfEnd));
auto *StartSCEV = SE.getConstant(Phi->getType(), 0);
auto *StepSCEV = SE.getConstant(Phi->getType(), 1);
EXPECT_EQ(Desc.getExpr(),
@@ -570,8 +568,6 @@ for.end:
auto &Chain = Desc.getChain();
EXPECT_TRUE(Chain.size() == 1 && Chain.contains(ChainPhi));
EXPECT_EQ(Desc.getStepInst(), StepInst);
- EXPECT_EQ(Desc.getPredicateEdge(),
- MonotonicDescriptor::Edge(IfThen, IfEnd));
auto *StartSCEV = SE.getSCEV(F.getArg(0));
auto *StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
EXPECT_EQ(Desc.getExpr(),
>From 0c4a91060924fef2448739e869728ba7d28588ce Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Tue, 18 Aug 2026 14:07:33 +0000
Subject: [PATCH 13/15] Rework
---
llvm/include/llvm/Analysis/IVDescriptors.h | 40 +++++++++++-----------
llvm/lib/Analysis/IVDescriptors.cpp | 30 ++++++++--------
2 files changed, 35 insertions(+), 35 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index a9995027c14fe..70c974c7a6c4c 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -491,21 +491,20 @@ class MonotonicDescriptor {
public:
MonotonicDescriptor() = default;
- MonotonicDescriptor(
- const SmallPtrSetImpl<PHINode *> &Chain,
- const DenseMap<Instruction *, const SCEV *> &CompressedMemOps,
- Instruction *StepInst, const SCEVAddRecExpr *Expr)
- : Chain(llvm::from_range, Chain), CompressedMemOps(CompressedMemOps),
+ MonotonicDescriptor(const SmallPtrSetImpl<PHINode *> &Chain,
+ const DenseMap<Value *, const SCEV *> &CompressedPtrs,
+ Instruction *StepInst, const SCEVAddRecExpr *Expr)
+ : Chain(llvm::from_range, Chain), CompressedPtrs(CompressedPtrs),
StepInst(StepInst), Expr(Expr) {}
/// Returns the PHIs that feed into the backedge of the monotonic PHI.
const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
- // Returns memory operations whose addresses are derived from this monotonic
- // PHI. The keys are load or store instructions, the values are SCEVAddRecs
- // that represent how the pointer is updated by StepInst.
- const DenseMap<Instruction *, const SCEV *> &getCompressedMemoryOps() const {
- return CompressedMemOps;
+ // Returns pointers (used by load/store operations) where the address is
+ // derived from this monotonic PHI. The keys are pointers, the values are
+ // SCEVAddRecs that represent how the pointer is updated by StepInst.
+ const DenseMap<Value *, const SCEV *> &getCompressedPtrs() const {
+ return CompressedPtrs;
}
/// Returns the instruction that updates the value of the monotonic PHI.
@@ -526,8 +525,8 @@ class MonotonicDescriptor {
/// The PHIs that feed into the backedge update of the monotonic PHI.
SmallPtrSet<PHINode *, 1> Chain;
- /// Memory operations whose addresses are derived from this monotonic PHI.
- DenseMap<Instruction *, const SCEV *> CompressedMemOps;
+ /// Pointers whose addresses are derived from this monotonic PHI.
+ DenseMap<Value *, const SCEV *> CompressedPtrs;
/// The instruction that updates the value of the monotonic PHI.
Instruction *StepInst = nullptr;
@@ -543,14 +542,15 @@ class MonotonicDescriptor {
SmallPtrSetImpl<PHINode *> &Chain,
ScalarEvolution &SE);
- /// Collects the memory operations whose addresses are derived from \p PN.
- /// The memory operations and SCEV expressions for their pointer operands are
- /// placed in \p CompressedMemOps. Returns true if no unexpected users of \p
- /// PN were found.
- static bool CollectCompressedMemOpUsers(
- PHINode *PN, const Loop *L, const SmallPtrSetImpl<PHINode *> &Chain,
- const SCEV *PhiSCEV, ScalarEvolution &SE,
- DenseMap<Instruction *, const SCEV *> &CompressedMemOps);
+ /// Collects pointers values (used by loads/stores) whose addresses are
+ /// derived from \p PN. The pointer operands and SCEV expressions for their
+ /// pointer operands are placed in \p CompressedPtrs. Returns true if no
+ /// unexpected users of \p PN were found.
+ static bool
+ CollectCompressedPointers(PHINode *PN, const Loop *L,
+ const SmallPtrSetImpl<PHINode *> &Chain,
+ const SCEV *PhiSCEV, ScalarEvolution &SE,
+ DenseMap<Value *, const SCEV *> &CompressedPtrs);
};
} // end namespace llvm
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 79c5feb6d5b22..582dbe632e997 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1789,22 +1789,22 @@ MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
return {StepInst, PhiSCEV};
}
-bool MonotonicDescriptor::CollectCompressedMemOpUsers(
+bool MonotonicDescriptor::CollectCompressedPointers(
PHINode *PN, const Loop *L, const SmallPtrSetImpl<PHINode *> &Chain,
const SCEV *PhiSCEV, ScalarEvolution &SE,
- DenseMap<Instruction *, const SCEV *> &CompressedMemOps) {
+ DenseMap<Value *, const SCEV *> &CompressedPtrs) {
ValueToSCEVMapTy PhiMap{{PN, PhiSCEV}};
- auto GetCompressedPtrSCEV = [&](Instruction *MemI) -> const SCEV * {
- const SCEV *PtrSCEV = SCEVParameterRewriter::rewrite(
- SE.getSCEV(getLoadStorePointerOperand(MemI)), SE, PhiMap);
+ auto GetCompressedPtrSCEV = [&](Value *Ptr, Type *AccessTy) -> const SCEV * {
+ const SCEV *PtrSCEV =
+ SCEVParameterRewriter::rewrite(SE.getSCEV(Ptr), SE, PhiMap);
auto *AddRec = dyn_cast<SCEVAddRecExpr>(PtrSCEV);
if (!AddRec || !AddRec->isAffine())
return nullptr;
// Check if pointer step equals access size.
SCEVUse Step = AddRec->getStepRecurrence(SE);
- if (Step != SE.getSizeOfExpr(Step->getType(), getLoadStoreType(MemI)))
+ if (Step != SE.getSizeOfExpr(Step->getType(), AccessTy))
return nullptr;
return PtrSCEV;
@@ -1827,10 +1827,11 @@ bool MonotonicDescriptor::CollectCompressedMemOpUsers(
SI && SI->getValueOperand() == CurrentVal)
return false;
- const SCEV *CompressedPtr = GetCompressedPtrSCEV(I);
- if (!CompressedPtr)
+ Value *Ptr = getLoadStorePointerOperand(I);
+ const SCEV *PrtSCEV = GetCompressedPtrSCEV(Ptr, getLoadStoreType(I));
+ if (!PrtSCEV)
return false;
- CompressedMemOps.insert({I, CompressedPtr});
+ CompressedPtrs.insert({Ptr, PrtSCEV});
continue;
}
@@ -1878,8 +1879,8 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
}
// Only allow the monotonic recurrence to feed the PHI chain or memory
- // operations addressed by a compressed pointer. Save those memory operations
- // on the descriptor.
+ // operations addressed by a compressed pointer. Save those pointers on the
+ // descriptor.
SmallPtrSet<PHINode *, 1> Chain;
auto [StepInst, PhiSCEV] =
CollectMonotonicPHIChain(PN, L, BackEdgeInst, Chain, SE);
@@ -1890,11 +1891,10 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
if (!PhiAddRec || !PhiAddRec->isAffine())
return false;
- DenseMap<Instruction *, const SCEV *> CompressedMemOps;
- if (!CollectCompressedMemOpUsers(PN, L, Chain, PhiAddRec, SE,
- CompressedMemOps))
+ DenseMap<Value *, const SCEV *> CompressedPtrs;
+ if (!CollectCompressedPointers(PN, L, Chain, PhiAddRec, SE, CompressedPtrs))
return false;
- Desc = MonotonicDescriptor(Chain, CompressedMemOps, StepInst, PhiAddRec);
+ Desc = MonotonicDescriptor(Chain, CompressedPtrs, StepInst, PhiAddRec);
return true;
}
>From cb411698058452d8b22a394fd50e5d12628451c2 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Tue, 18 Aug 2026 16:37:54 +0000
Subject: [PATCH 14/15] Fixups
---
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index 28c563830e172..ce8a7d2d465b9 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -502,9 +502,9 @@ for.end:
EXPECT_EQ(Desc.getExpr(),
SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
// Check %arrayidx use.
- EXPECT_EQ(Desc.getCompressedMemoryOps().size(), 1U);
- auto [StoreInst, PtrSCEV] = *Desc.getCompressedMemoryOps().begin();
- EXPECT_EQ(getPointerOperand(StoreInst), GEPInst);
+ EXPECT_EQ(Desc.getCompressedPtrs().size(), 1U);
+ auto [StorePtr, PtrSCEV] = *Desc.getCompressedPtrs().begin();
+ EXPECT_EQ(StorePtr, GEPInst);
StartSCEV = SE.getSCEV(F.getArg(0));
StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
EXPECT_EQ(PtrSCEV,
>From 34898bbdbffce2967a1f01a3b2dc339b1eb00316 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Tue, 18 Aug 2026 16:54:40 +0000
Subject: [PATCH 15/15] Fixups
---
llvm/include/llvm/Analysis/IVDescriptors.h | 14 +++++++-------
llvm/lib/Analysis/IVDescriptors.cpp | 8 ++++----
2 files changed, 11 insertions(+), 11 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 70c974c7a6c4c..b555cfd7d722c 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -538,19 +538,19 @@ class MonotonicDescriptor {
/// Verifies \p PN is a monotonic PHI and collects the PHIs within the chain.
/// Returns the StepInst and a SCEV expression for \p PN.
static std::pair<Instruction *, const SCEV *>
- CollectMonotonicPHIChain(PHINode *PN, const Loop *L, PHINode *BackEdgeInst,
+ collectMonotonicPHIChain(PHINode *PN, const Loop *L, PHINode *BackEdgeInst,
SmallPtrSetImpl<PHINode *> &Chain,
ScalarEvolution &SE);
/// Collects pointers values (used by loads/stores) whose addresses are
/// derived from \p PN. The pointer operands and SCEV expressions for their
- /// pointer operands are placed in \p CompressedPtrs. Returns true if no
- /// unexpected users of \p PN were found.
+ /// pointer operands are placed in \p CompressedPtrs. Returns true if all
+ /// users of \p PN were supported memory operations.
static bool
- CollectCompressedPointers(PHINode *PN, const Loop *L,
- const SmallPtrSetImpl<PHINode *> &Chain,
- const SCEV *PhiSCEV, ScalarEvolution &SE,
- DenseMap<Value *, const SCEV *> &CompressedPtrs);
+ collectAllowedMemoryUses(PHINode *PN, const Loop *L,
+ const SmallPtrSetImpl<PHINode *> &Chain,
+ const SCEV *PhiSCEV, ScalarEvolution &SE,
+ DenseMap<Value *, const SCEV *> &CompressedPtrs);
};
} // end namespace llvm
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 582dbe632e997..8bc17bef1ea39 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1727,7 +1727,7 @@ bool InductionDescriptor::isInductionPHI(
// For this pattern, monotonic phi is described by {%start, +, %step}
// recurrence and predicate is CFG edge %step_bb -> %bbN.
std::pair<Instruction *, const SCEV *>
-MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
+MonotonicDescriptor::collectMonotonicPHIChain(PHINode *PN, const Loop *L,
PHINode *BackEdgeInst,
SmallPtrSetImpl<PHINode *> &Chain,
ScalarEvolution &SE) {
@@ -1789,7 +1789,7 @@ MonotonicDescriptor::CollectMonotonicPHIChain(PHINode *PN, const Loop *L,
return {StepInst, PhiSCEV};
}
-bool MonotonicDescriptor::CollectCompressedPointers(
+bool MonotonicDescriptor::collectAllowedMemoryUses(
PHINode *PN, const Loop *L, const SmallPtrSetImpl<PHINode *> &Chain,
const SCEV *PhiSCEV, ScalarEvolution &SE,
DenseMap<Value *, const SCEV *> &CompressedPtrs) {
@@ -1883,7 +1883,7 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
// descriptor.
SmallPtrSet<PHINode *, 1> Chain;
auto [StepInst, PhiSCEV] =
- CollectMonotonicPHIChain(PN, L, BackEdgeInst, Chain, SE);
+ collectMonotonicPHIChain(PN, L, BackEdgeInst, Chain, SE);
if (!StepInst)
return false;
@@ -1892,7 +1892,7 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
return false;
DenseMap<Value *, const SCEV *> CompressedPtrs;
- if (!CollectCompressedPointers(PN, L, Chain, PhiAddRec, SE, CompressedPtrs))
+ if (!collectAllowedMemoryUses(PN, L, Chain, PhiAddRec, SE, CompressedPtrs))
return false;
Desc = MonotonicDescriptor(Chain, CompressedPtrs, StepInst, PhiAddRec);
More information about the llvm-commits
mailing list