[llvm] [IVDescriptors] Implement MonotonicDescriptor (PR #214490)
Benjamin Maxwell via llvm-commits
llvm-commits at lists.llvm.org
Thu Aug 13 02:00:34 PDT 2026
https://github.com/MacDue updated https://github.com/llvm/llvm-project/pull/214490
>From 398841f255b48dcc97a051137c7b02bcd7ea5a74 Mon Sep 17 00:00:00 2001
From: Sergey Kachkov <sergey.kachkov at syntacore.com>
Date: Wed, 29 Jan 2025 15:27:01 +0300
Subject: [PATCH 1/5] [IVDescriptors] Implement MonotonicDescriptor
RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442
"Monotonic" variable is similar to induction variable, but its value is updated under some condition, e.g.:
```
int idx = 0;
for(int i = 0; i < n; ++i) {
// some uses of idx
if (cond)
++idx;
}
```
In this example, `i` is induction variable and `idx` is monotonic variable: it's updated only when cond == true. In LLVM IR, this looks like:
```
loop_header:
%monotonic_phi = [%start, %prehader], [ %chain_phi0, %latch]
step_bb:
%step = add/gep %monotonic_phi, %step_val
bbN:
%chain_phiN = [%step, %step_bb], [%monotonic_phi, %some_pred]
...
bb1:
%chain_phi1 = [%monotonic_phi, %some_pred], [%chain_phi2, %some_pred]
loop_latch:
%chain_phi0 = [%monotonic_phi, %some_pred], [%chain_phi1, %some_pred]
```
We start the analysis from the header phi (%monotonic_phi). Its backedge value (%chain_phi0) can be either unmodified (%monotonic_val) or the incremented one (%step) depending on some condition. Therefore, all incoming values in chain phi should be %monotonic_phi except the one that points to the next chain phi or the step instruction. The analysis stops when the correct step instruction is found (correct step instruction is in form add/gep %monotonic_phi, %step_val, and %step_val is loop-invariant). In this way, %monotonic_val is incremented on %step_val only when last chain phi "selects" %step incoming value; in other words, when step_bb -> bbN edge is executed. The condition of monotonic variable update is stored as such CFG edge (`getPredicateEdge()` method). The other MonotonicDescriptor methods are:
1. `getChain()` - returns set of chain phis (%chain_phi0, ..., %chain_phiN)
2. `getStepInst()` - returns %step instruction (either add or getlementptr)
3. `getExpr()` - returns SCEVAddRec in assumption that condition of update is always true (SCEVAddRec + condtion predicate fully describes evolution of monotonic variable in the loop)
MonotonicDescriptor also implements `isMonotonicVal` method. This routine will allow to recognize values which have monotonic phi as a transitive dependency, e.g. array subscript operators `arr[idx]`, where idx is monotonic phi. The descriptor of monotonic value is equivalent to the descriptor of monotonic phi, except its SCEV: we obtain the SCEV of monotonic value and replace the SCEVUnknown expression that corresponds to monotonic phi with the SCEVAddRec from monotonic phi descriptor. The resulted SCEV shows the evolution of monotonic value in loop iterations when predicate is true (similarly to SCEV).
*Restrictions*
1. We only support "post-increment" update of monotonic vars; in other words, uses of chain phis (other than in other chain phis) are prohibited. It's only alowed to use %monotonic_phi outside of described chain pattern.
2. We don't support `select` instructions in monotonic patterns (instead of chain phis). In theory this can be implemented, but on practice we want to use MonotonicDescriptor to recognize expandloads/compressstores, where step instruction is placed near memory instruction, so if-conversion to select is not applied there (because basic block can't be eliminated anyway). So, this restriction doesn't limit our abilities to recognize expandloads/compressstore patterns.
This is a continuation Sergey Kachkov's patch (#140720).
---
llvm/include/llvm/Analysis/IVDescriptors.h | 39 +++++
llvm/lib/Analysis/IVDescriptors.cpp | 121 ++++++++++++++
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 153 ++++++++++++++++++
3 files changed, 313 insertions(+)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index bad372421dfae..c644bac9a3ecc 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -29,6 +29,7 @@ class PredicatedScalarEvolution;
class ScalarEvolution;
class SCEV;
class SCEVPredicate;
+class SCEVAddRecExpr;
class StoreInst;
/// These are the kinds of recurrences that we support.
@@ -482,6 +483,44 @@ class InductionDescriptor {
SmallVector<const SCEVPredicate *, 2> NoWrapPredicates;
};
+/// A struct for saving information about monotonic variables.
+/// Monotonic variable can be considered as a "conditional" induction variable:
+/// its update happens only on loop iterations for which a certain predicate is
+/// satisfied. In this implementation the predicate is represented as an edge in
+/// loop CFG: variable is updated if this edge is executed on current loop
+/// iteration.
+class MonotonicDescriptor {
+public:
+ using Edge = std::pair<BasicBlock *, BasicBlock *>;
+
+ MonotonicDescriptor() = default;
+
+ const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
+ Instruction *getStepInst() const { return StepInst; }
+ Edge getPredicateEdge() const { return PredEdge; }
+ const SCEVAddRecExpr *getExpr() const { return Expr; }
+
+ /// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
+ /// is monotonic, the monotonic descriptor \p D will contain the data
+ /// describing this variable.
+ static bool isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+ /// Returns true if \p Val is a monotonic variable in the loop \p L (in this
+ /// case, the value should transitively contain monotonic phi as part of its
+ /// calculation).
+ static bool isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+private:
+ SmallPtrSet<PHINode *, 1> Chain;
+ Instruction *StepInst;
+ Edge PredEdge;
+ const SCEVAddRecExpr *Expr;
+
+ bool setSCEV(const SCEV *NewExpr);
+};
+
} // end namespace llvm
#endif // LLVM_ANALYSIS_IVDESCRIPTORS_H
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 800b64e9a29af..595e8b6dc1f78 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1696,3 +1696,124 @@ bool InductionDescriptor::isInductionPHI(
/*InductionBinOp=*/nullptr, /*Casts=*/nullptr, Preds);
return true;
}
+
+bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
+ auto *AddRec = dyn_cast<SCEVAddRecExpr>(NewExpr);
+ if (!AddRec || !AddRec->isAffine())
+ return false;
+ Expr = AddRec;
+ return true;
+}
+
+// Recognize monotonic phi variable by matching the following pattern:
+// loop_header:
+// %monotonic_phi = [%start, %preheader], [%chain_phi0, %latch]
+//
+// step_bb:
+// %step = add/gep %monotonic_phi, %step_val
+//
+// bbN:
+// %chain_phiN = [%monotonic_phi, ], [%step, ]
+//
+// ...
+//
+// bb1:
+// %chain_phi1 = [%monotonic_phi, ], [%chain_phi2, ]
+//
+// latch:
+// %chain_phi0 = [%monotonic_phi, %pred], [%chain_phi1, %pred]
+//
+// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
+// and predicate is CFG edge %step_bb -> %bbN.
+bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!PN->getType()->isIntOrPtrTy() || PN->getParent() != L->getHeader())
+ return false;
+ auto *BackEdgeInst =
+ dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
+ if (!BackEdgeInst)
+ return false;
+ PHINode *PHIChain = BackEdgeInst;
+ std::optional<std::pair<Edge, Value *>> Inc;
+ while (PHIChain) {
+ Desc.Chain.insert(PHIChain);
+ PHINode *NextPHIChain = nullptr;
+ for (auto [Block, Incoming] :
+ zip_equal(PHIChain->blocks(), PHIChain->incoming_values())) {
+ if (Incoming == PN)
+ continue;
+ if (!Incoming->hasOneUse())
+ return false;
+ if (auto *IncomingPHI = dyn_cast<PHINode>(Incoming)) {
+ if (NextPHIChain)
+ return false;
+ NextPHIChain = IncomingPHI;
+ continue;
+ }
+ if (Inc || NextPHIChain)
+ return false;
+ Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
+ }
+ PHIChain = NextPHIChain;
+ }
+ if (!Inc)
+ return false;
+ auto [PredEdge, StepOp] = *Inc;
+ auto *StepInst = dyn_cast<Instruction>(StepOp);
+ if (!StepInst)
+ return false;
+ Desc.StepInst = StepInst;
+ Desc.PredEdge = PredEdge;
+
+ // Construct SCEVAddRec for this value.
+ Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
+
+ Value *Step = nullptr;
+ bool StepMatch =
+ PN->getType()->isPointerTy()
+ ? match(StepInst, m_PtrAdd(m_Specific(PN), m_Value(Step)))
+ : match(StepInst, m_Add(m_Specific(PN), m_Value(Step)));
+ if (!StepMatch || !L->isLoopInvariant(Step))
+ return false;
+
+ SCEV::NoWrapFlags WrapFlags = SCEV::FlagAnyWrap;
+ if (auto *GEP = dyn_cast<GEPOperator>(StepInst)) {
+ if (GEP->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (GEP->hasNoUnsignedSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ } else if (auto *OBO = dyn_cast<OverflowingBinaryOperator>(StepInst)) {
+ if (OBO->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (OBO->hasNoSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ }
+
+ return Desc.setSCEV(
+ SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags));
+}
+
+bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!Val->getType()->isIntOrPtrTy() || L->isLoopInvariant(Val))
+ return false;
+ auto *CurInst = cast<Instruction>(Val);
+
+ auto LoopVariantVal = [&](Value *V, bool AllowRepeats) {
+ return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
+ };
+
+ while (!isa<PHINode>(CurInst)) {
+ CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
+ if (!CurInst)
+ return false;
+ };
+
+ if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
+ return false;
+
+ ValueToSCEVMapTy Map{{CurInst, Desc.getExpr()}};
+ return Desc.setSCEV(SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
+}
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index faf30fd322c10..753abb0c7b93d 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -10,6 +10,7 @@
#include "llvm/Analysis/AssumptionCache.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/AsmParser/Parser.h"
#include "llvm/IR/Dominators.h"
@@ -430,3 +431,155 @@ TEST(IVDescriptorsTest, InvariantStoreNoSCEV) {
EXPECT_FALSE(IsRdxPhi);
});
}
+
+TEST(IVDescriptorsTest, MonotonicIntVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %dst, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi i32 [ 0, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = add nsw i32 %monotonic, 1
+ %monotonic.prom = sext i32 %monotonic to i64
+ %arrayidx = getelementptr inbounds i32, ptr %dst, i64 %monotonic.prom
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi i32 [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ BBI = IfThen->begin();
+ Instruction *StepInst = &*BBI;
+ assert(StepInst->getName() == "inc");
+ Instruction *ExtInst = &*(++BBI);
+ assert(ExtInst->getName() == "monotonic.prom");
+ Instruction *GEPInst = &*(++BBI);
+ assert(GEPInst->getName() == "arrayidx");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ // Check %monotonic descriptor.
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &PhiChain = Desc.getChain();
+ EXPECT_TRUE(PhiChain.size() == 1 && PhiChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getConstant(Phi->getType(), 0);
+ auto *StepSCEV = SE.getConstant(Phi->getType(), 1);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ // Check %arrayidx descriptor.
+ bool IsMonotonicVal =
+ MonotonicDescriptor::isMonotonicVal(GEPInst, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicVal);
+ // Chain, StepInst and PredicateEdge are the same with %monotonic.
+ auto &ValChain = Desc.getChain();
+ EXPECT_TRUE(ValChain.size() == 1 && ValChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ StartSCEV = SE.getSCEV(F.getArg(0));
+ StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
+
+TEST(IVDescriptorsTest, MonotonicPtrVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %start, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi ptr [ %start, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = getelementptr inbounds i8, ptr %monotonic, i64 4
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi ptr [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ Instruction *StepInst = &*(IfThen->begin());
+ assert(StepInst->getName() == "inc");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &Chain = Desc.getChain();
+ EXPECT_TRUE(Chain.size() == 1 && Chain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getSCEV(F.getArg(0));
+ auto *StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
>From 9765af1faab82a63f20c76190c12f9a81dc51cea Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Thu, 6 Aug 2026 19:31:23 +0000
Subject: [PATCH 2/5] remove ;
---
llvm/lib/Analysis/IVDescriptors.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 595e8b6dc1f78..db1ace2209b2c 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1809,7 +1809,7 @@ bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
if (!CurInst)
return false;
- };
+ }
if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
return false;
>From 1afeb6f742b35abe70074953e11086405ff002ee Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Fri, 7 Aug 2026 09:17:28 +0000
Subject: [PATCH 3/5] Add docs
---
llvm/include/llvm/Analysis/IVDescriptors.h | 31 +++++++++++++++++++---
1 file changed, 27 insertions(+), 4 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index c644bac9a3ecc..4a1aa5830d51c 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -495,9 +495,19 @@ class MonotonicDescriptor {
MonotonicDescriptor() = default;
+ /// Returns the PHIs that feed into the backedge of the monotonic PHI.
const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
+
+ /// Returns the instruction that updates the value of the monotonic PHI.
Instruction *getStepInst() const { return StepInst; }
+
+ /// Returns the edge where the monotonic value/PHI is updated when taken.
Edge getPredicateEdge() const { return PredEdge; }
+
+ /// Returns the expression that represents the monotonic PHI (match with
+ /// isMonotonicPHI) or monotonic value (match with isMonotonicVal). Note: The
+ /// conditional update is represented with a plain SCEVAddRec within the
+ /// expression, this only holds along the predicated edge.
const SCEVAddRecExpr *getExpr() const { return Expr; }
/// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
@@ -507,17 +517,30 @@ class MonotonicDescriptor {
MonotonicDescriptor &Desc, ScalarEvolution &SE);
/// Returns true if \p Val is a monotonic variable in the loop \p L (in this
- /// case, the value should transitively contain monotonic phi as part of its
+ /// case, the value should transitively contain monotonic PHI as part of its
/// calculation).
static bool isMonotonicVal(Value *Val, const Loop *L,
MonotonicDescriptor &Desc, ScalarEvolution &SE);
private:
+ /// The PHIs that feed into the backedge update of the monotonic PHI.
SmallPtrSet<PHINode *, 1> Chain;
- Instruction *StepInst;
- Edge PredEdge;
- const SCEVAddRecExpr *Expr;
+ /// The instruction that updates the value of the monotonic PHI.
+ Instruction *StepInst = nullptr;
+
+ /// The predicated edge where the monotonic value/PHI is updated.
+ /// The common case is {StepInstBlock, StepInstBlock->getSingleSuccessor()}.
+ /// Note: StepInstBlock = StepInst->getParent().
+ Edge PredEdge = {};
+
+ /// Expression that represents the monotonic PHI (isMonotonicPHI) or monotonic
+ /// (isMonotonicVal). Within the expression, the conditional update is
+ /// represented as an (unconditional) SCEVAddRec.
+ const SCEVAddRecExpr *Expr = nullptr;
+
+ /// Set the SCEV expression for this descriptor. \p NewExpr must be an affine
+ /// SCEVAddRec.
bool setSCEV(const SCEV *NewExpr);
};
>From 0206e68fa33a24f2fe6fcdc392c69e48150f7084 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Mon, 10 Aug 2026 09:18:28 +0000
Subject: [PATCH 4/5] Fixups
---
llvm/include/llvm/Analysis/IVDescriptors.h | 9 +++++----
llvm/lib/Analysis/IVDescriptors.cpp | 3 ++-
2 files changed, 7 insertions(+), 5 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 4a1aa5830d51c..0d7e48e441f72 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -517,8 +517,8 @@ class MonotonicDescriptor {
MonotonicDescriptor &Desc, ScalarEvolution &SE);
/// Returns true if \p Val is a monotonic variable in the loop \p L (in this
- /// case, the value should transitively contain monotonic PHI as part of its
- /// calculation).
+ /// case, the value should transitively contain monotonic PHI as the only
+ /// loop-variant part/operand of its calculation).
static bool isMonotonicVal(Value *Val, const Loop *L,
MonotonicDescriptor &Desc, ScalarEvolution &SE);
@@ -539,8 +539,9 @@ class MonotonicDescriptor {
/// represented as an (unconditional) SCEVAddRec.
const SCEVAddRecExpr *Expr = nullptr;
- /// Set the SCEV expression for this descriptor. \p NewExpr must be an affine
- /// SCEVAddRec.
+ /// Set the SCEV expression for this descriptor. Returns true if the
+ /// expression was successfully updated, this requires \p NewExpr must be an
+ /// affine SCEVAddRec.
bool setSCEV(const SCEV *NewExpr);
};
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index db1ace2209b2c..ea7da3e89f5b4 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1751,6 +1751,7 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
NextPHIChain = IncomingPHI;
continue;
}
+ // Only one update/step is allowed. The unmodified value must be PN.
if (Inc || NextPHIChain)
return false;
Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
@@ -1801,7 +1802,7 @@ bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
return false;
auto *CurInst = cast<Instruction>(Val);
- auto LoopVariantVal = [&](Value *V, bool AllowRepeats) {
+ auto LoopVariantVal = [&](Value *V, bool /*AllowRepeats*/) {
return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
};
>From 73f42560e637432b9ade026919dc07e77b5764a0 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Tue, 11 Aug 2026 12:16:42 +0000
Subject: [PATCH 5/5] Fixups
---
llvm/include/llvm/Analysis/IVDescriptors.h | 6 +++
llvm/lib/Analysis/IVDescriptors.cpp | 45 ++++++++++++++--------
2 files changed, 35 insertions(+), 16 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index 0d7e48e441f72..081a4412550da 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -495,6 +495,12 @@ class MonotonicDescriptor {
MonotonicDescriptor() = default;
+ MonotonicDescriptor(SmallPtrSetImpl<PHINode *> &Chain, Instruction *StepInst,
+ Edge PredEdge, const SCEV *Expr)
+ : Chain(llvm::from_range, Chain), StepInst(StepInst), PredEdge(PredEdge) {
+ setSCEV(Expr);
+ }
+
/// Returns the PHIs that feed into the backedge of the monotonic PHI.
const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index ea7da3e89f5b4..8e49666bf7415 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1707,21 +1707,30 @@ bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
// Recognize monotonic phi variable by matching the following pattern:
// loop_header:
-// %monotonic_phi = [%start, %preheader], [%chain_phi0, %latch]
+// %monotonic_phi = phi [ %start, %preheader ], [ %chain_phi0, %latch ]
+// br i1 %do_step, label %step_bb, label %bbN
//
// step_bb:
// %step = add/gep %monotonic_phi, %step_val
+// br label %bbN
//
// bbN:
-// %chain_phiN = [%monotonic_phi, ], [%step, ]
+// %chain_phiN = phi [ %monotonic_phi, %loop_header ], [ %step, %step_bb ]
+// br label %bb(N-1)
//
// ...
//
+// bb2:
+// %chain_phi2 = phi [ %monotonic_phi, %pred2 ], [ %chain_phi3, %bb3 ]
+// br label %bb1
+//
// bb1:
-// %chain_phi1 = [%monotonic_phi, ], [%chain_phi2, ]
+// %chain_phi1 = phi [ %monotonic_phi, %pred1 ], [ %chain_phi2, %bb2 ]
+// br label %latch
//
// latch:
-// %chain_phi0 = [%monotonic_phi, %pred], [%chain_phi1, %pred]
+// %chain_phi0 = phi [ %monotonic_phi, %pred0 ], [ %chain_phi1, %bb1 ]
+// br label %loop_header
//
// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
// and predicate is CFG edge %step_bb -> %bbN.
@@ -1734,10 +1743,14 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
if (!BackEdgeInst)
return false;
+
+ Edge PredEdge;
+ Value *StepOp = nullptr;
+ SmallPtrSet<PHINode *, 1> Chain;
PHINode *PHIChain = BackEdgeInst;
- std::optional<std::pair<Edge, Value *>> Inc;
+
while (PHIChain) {
- Desc.Chain.insert(PHIChain);
+ Chain.insert(PHIChain);
PHINode *NextPHIChain = nullptr;
for (auto [Block, Incoming] :
zip_equal(PHIChain->blocks(), PHIChain->incoming_values())) {
@@ -1752,20 +1765,17 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
continue;
}
// Only one update/step is allowed. The unmodified value must be PN.
- if (Inc || NextPHIChain)
+ if (StepOp || NextPHIChain)
return false;
- Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
+ PredEdge = Edge{Block, PHIChain->getParent()};
+ StepOp = Incoming;
}
PHIChain = NextPHIChain;
}
- if (!Inc)
- return false;
- auto [PredEdge, StepOp] = *Inc;
- auto *StepInst = dyn_cast<Instruction>(StepOp);
+
+ auto *StepInst = dyn_cast_if_present<Instruction>(StepOp);
if (!StepInst)
return false;
- Desc.StepInst = StepInst;
- Desc.PredEdge = PredEdge;
// Construct SCEVAddRec for this value.
Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
@@ -1791,8 +1801,11 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
}
- return Desc.setSCEV(
- SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags));
+ const SCEV *PhiSCEV =
+ SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags);
+
+ Desc = MonotonicDescriptor(Chain, StepInst, PredEdge, PhiSCEV);
+ return Desc.getExpr() != nullptr;
}
bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
More information about the llvm-commits
mailing list