[llvm] [IVDescriptors] Implement MonotonicDescriptor (PR #214490)
Benjamin Maxwell via llvm-commits
llvm-commits at lists.llvm.org
Fri Aug 21 08:51:51 PDT 2026
https://github.com/MacDue updated https://github.com/llvm/llvm-project/pull/214490
>From bb7ca948939d269254350d77a76e9b676831f717 Mon Sep 17 00:00:00 2001
From: Sergey Kachkov <sergey.kachkov at syntacore.com>
Date: Wed, 29 Jan 2025 15:27:01 +0300
Subject: [PATCH 1/3] [IVDescriptors] Implement MonotonicDescriptor
RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442
"Monotonic" variable is similar to induction variable, but its value is updated under some condition, e.g.:
```
int idx = 0;
for(int i = 0; i < n; ++i) {
// some uses of idx
if (cond)
++idx;
}
```
In this example, `i` is induction variable and `idx` is monotonic variable: it's updated only when cond == true. In LLVM IR, this looks like:
```
loop_header:
%monotonic_phi = [%start, %prehader], [ %chain_phi0, %latch]
step_bb:
%step = add/gep %monotonic_phi, %step_val
bbN:
%chain_phiN = [%step, %step_bb], [%monotonic_phi, %some_pred]
...
bb1:
%chain_phi1 = [%monotonic_phi, %some_pred], [%chain_phi2, %some_pred]
loop_latch:
%chain_phi0 = [%monotonic_phi, %some_pred], [%chain_phi1, %some_pred]
```
We start the analysis from the header phi (%monotonic_phi). Its backedge value (%chain_phi0) can be either unmodified (%monotonic_val) or the incremented one (%step) depending on some condition. Therefore, all incoming values in chain phi should be %monotonic_phi except the one that points to the next chain phi or the step instruction. The analysis stops when the correct step instruction is found (correct step instruction is in form add/gep %monotonic_phi, %step_val, and %step_val is loop-invariant). In this way, %monotonic_val is incremented on %step_val only when last chain phi "selects" %step incoming value; in other words, when step_bb -> bbN edge is executed. The condition of monotonic variable update is stored as such CFG edge (`getPredicateEdge()` method). The other MonotonicDescriptor methods are:
1. `getChain()` - returns set of chain phis (%chain_phi0, ..., %chain_phiN)
2. `getStepInst()` - returns %step instruction (either add or getlementptr)
3. `getExpr()` - returns SCEVAddRec in assumption that condition of update is always true (SCEVAddRec + condtion predicate fully describes evolution of monotonic variable in the loop)
MonotonicDescriptor also implements `isMonotonicVal` method. This routine will allow to recognize values which have monotonic phi as a transitive dependency, e.g. array subscript operators `arr[idx]`, where idx is monotonic phi. The descriptor of monotonic value is equivalent to the descriptor of monotonic phi, except its SCEV: we obtain the SCEV of monotonic value and replace the SCEVUnknown expression that corresponds to monotonic phi with the SCEVAddRec from monotonic phi descriptor. The resulted SCEV shows the evolution of monotonic value in loop iterations when predicate is true (similarly to SCEV).
*Restrictions*
1. We only support "post-increment" update of monotonic vars; in other words, uses of chain phis (other than in other chain phis) are prohibited. It's only alowed to use %monotonic_phi outside of described chain pattern.
2. We don't support `select` instructions in monotonic patterns (instead of chain phis). In theory this can be implemented, but on practice we want to use MonotonicDescriptor to recognize expandloads/compressstores, where step instruction is placed near memory instruction, so if-conversion to select is not applied there (because basic block can't be eliminated anyway). So, this restriction doesn't limit our abilities to recognize expandloads/compressstore patterns.
This is a continuation Sergey Kachkov's patch (#140720).
---
llvm/include/llvm/Analysis/IVDescriptors.h | 39 +++++
llvm/lib/Analysis/IVDescriptors.cpp | 121 ++++++++++++++
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 153 ++++++++++++++++++
3 files changed, 313 insertions(+)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index bad372421dfae..c644bac9a3ecc 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -29,6 +29,7 @@ class PredicatedScalarEvolution;
class ScalarEvolution;
class SCEV;
class SCEVPredicate;
+class SCEVAddRecExpr;
class StoreInst;
/// These are the kinds of recurrences that we support.
@@ -482,6 +483,44 @@ class InductionDescriptor {
SmallVector<const SCEVPredicate *, 2> NoWrapPredicates;
};
+/// A struct for saving information about monotonic variables.
+/// Monotonic variable can be considered as a "conditional" induction variable:
+/// its update happens only on loop iterations for which a certain predicate is
+/// satisfied. In this implementation the predicate is represented as an edge in
+/// loop CFG: variable is updated if this edge is executed on current loop
+/// iteration.
+class MonotonicDescriptor {
+public:
+ using Edge = std::pair<BasicBlock *, BasicBlock *>;
+
+ MonotonicDescriptor() = default;
+
+ const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
+ Instruction *getStepInst() const { return StepInst; }
+ Edge getPredicateEdge() const { return PredEdge; }
+ const SCEVAddRecExpr *getExpr() const { return Expr; }
+
+ /// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
+ /// is monotonic, the monotonic descriptor \p D will contain the data
+ /// describing this variable.
+ static bool isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+ /// Returns true if \p Val is a monotonic variable in the loop \p L (in this
+ /// case, the value should transitively contain monotonic phi as part of its
+ /// calculation).
+ static bool isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+private:
+ SmallPtrSet<PHINode *, 1> Chain;
+ Instruction *StepInst;
+ Edge PredEdge;
+ const SCEVAddRecExpr *Expr;
+
+ bool setSCEV(const SCEV *NewExpr);
+};
+
} // end namespace llvm
#endif // LLVM_ANALYSIS_IVDESCRIPTORS_H
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 800b64e9a29af..595e8b6dc1f78 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1696,3 +1696,124 @@ bool InductionDescriptor::isInductionPHI(
/*InductionBinOp=*/nullptr, /*Casts=*/nullptr, Preds);
return true;
}
+
+bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
+ auto *AddRec = dyn_cast<SCEVAddRecExpr>(NewExpr);
+ if (!AddRec || !AddRec->isAffine())
+ return false;
+ Expr = AddRec;
+ return true;
+}
+
+// Recognize monotonic phi variable by matching the following pattern:
+// loop_header:
+// %monotonic_phi = [%start, %preheader], [%chain_phi0, %latch]
+//
+// step_bb:
+// %step = add/gep %monotonic_phi, %step_val
+//
+// bbN:
+// %chain_phiN = [%monotonic_phi, ], [%step, ]
+//
+// ...
+//
+// bb1:
+// %chain_phi1 = [%monotonic_phi, ], [%chain_phi2, ]
+//
+// latch:
+// %chain_phi0 = [%monotonic_phi, %pred], [%chain_phi1, %pred]
+//
+// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
+// and predicate is CFG edge %step_bb -> %bbN.
+bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!PN->getType()->isIntOrPtrTy() || PN->getParent() != L->getHeader())
+ return false;
+ auto *BackEdgeInst =
+ dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
+ if (!BackEdgeInst)
+ return false;
+ PHINode *PHIChain = BackEdgeInst;
+ std::optional<std::pair<Edge, Value *>> Inc;
+ while (PHIChain) {
+ Desc.Chain.insert(PHIChain);
+ PHINode *NextPHIChain = nullptr;
+ for (auto [Block, Incoming] :
+ zip_equal(PHIChain->blocks(), PHIChain->incoming_values())) {
+ if (Incoming == PN)
+ continue;
+ if (!Incoming->hasOneUse())
+ return false;
+ if (auto *IncomingPHI = dyn_cast<PHINode>(Incoming)) {
+ if (NextPHIChain)
+ return false;
+ NextPHIChain = IncomingPHI;
+ continue;
+ }
+ if (Inc || NextPHIChain)
+ return false;
+ Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
+ }
+ PHIChain = NextPHIChain;
+ }
+ if (!Inc)
+ return false;
+ auto [PredEdge, StepOp] = *Inc;
+ auto *StepInst = dyn_cast<Instruction>(StepOp);
+ if (!StepInst)
+ return false;
+ Desc.StepInst = StepInst;
+ Desc.PredEdge = PredEdge;
+
+ // Construct SCEVAddRec for this value.
+ Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
+
+ Value *Step = nullptr;
+ bool StepMatch =
+ PN->getType()->isPointerTy()
+ ? match(StepInst, m_PtrAdd(m_Specific(PN), m_Value(Step)))
+ : match(StepInst, m_Add(m_Specific(PN), m_Value(Step)));
+ if (!StepMatch || !L->isLoopInvariant(Step))
+ return false;
+
+ SCEV::NoWrapFlags WrapFlags = SCEV::FlagAnyWrap;
+ if (auto *GEP = dyn_cast<GEPOperator>(StepInst)) {
+ if (GEP->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (GEP->hasNoUnsignedSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ } else if (auto *OBO = dyn_cast<OverflowingBinaryOperator>(StepInst)) {
+ if (OBO->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (OBO->hasNoSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ }
+
+ return Desc.setSCEV(
+ SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags));
+}
+
+bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!Val->getType()->isIntOrPtrTy() || L->isLoopInvariant(Val))
+ return false;
+ auto *CurInst = cast<Instruction>(Val);
+
+ auto LoopVariantVal = [&](Value *V, bool AllowRepeats) {
+ return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
+ };
+
+ while (!isa<PHINode>(CurInst)) {
+ CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
+ if (!CurInst)
+ return false;
+ };
+
+ if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
+ return false;
+
+ ValueToSCEVMapTy Map{{CurInst, Desc.getExpr()}};
+ return Desc.setSCEV(SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
+}
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index faf30fd322c10..753abb0c7b93d 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -10,6 +10,7 @@
#include "llvm/Analysis/AssumptionCache.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/AsmParser/Parser.h"
#include "llvm/IR/Dominators.h"
@@ -430,3 +431,155 @@ TEST(IVDescriptorsTest, InvariantStoreNoSCEV) {
EXPECT_FALSE(IsRdxPhi);
});
}
+
+TEST(IVDescriptorsTest, MonotonicIntVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %dst, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi i32 [ 0, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = add nsw i32 %monotonic, 1
+ %monotonic.prom = sext i32 %monotonic to i64
+ %arrayidx = getelementptr inbounds i32, ptr %dst, i64 %monotonic.prom
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi i32 [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ BBI = IfThen->begin();
+ Instruction *StepInst = &*BBI;
+ assert(StepInst->getName() == "inc");
+ Instruction *ExtInst = &*(++BBI);
+ assert(ExtInst->getName() == "monotonic.prom");
+ Instruction *GEPInst = &*(++BBI);
+ assert(GEPInst->getName() == "arrayidx");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ // Check %monotonic descriptor.
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &PhiChain = Desc.getChain();
+ EXPECT_TRUE(PhiChain.size() == 1 && PhiChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getConstant(Phi->getType(), 0);
+ auto *StepSCEV = SE.getConstant(Phi->getType(), 1);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ // Check %arrayidx descriptor.
+ bool IsMonotonicVal =
+ MonotonicDescriptor::isMonotonicVal(GEPInst, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicVal);
+ // Chain, StepInst and PredicateEdge are the same with %monotonic.
+ auto &ValChain = Desc.getChain();
+ EXPECT_TRUE(ValChain.size() == 1 && ValChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ StartSCEV = SE.getSCEV(F.getArg(0));
+ StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
+
+TEST(IVDescriptorsTest, MonotonicPtrVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %start, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi ptr [ %start, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = getelementptr inbounds i8, ptr %monotonic, i64 4
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi ptr [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ Instruction *StepInst = &*(IfThen->begin());
+ assert(StepInst->getName() == "inc");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &Chain = Desc.getChain();
+ EXPECT_TRUE(Chain.size() == 1 && Chain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getSCEV(F.getArg(0));
+ auto *StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
>From ae84e7d1c4a15cf28fafd8d3d266b139924f2295 Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Fri, 21 Aug 2026 14:34:40 +0000
Subject: [PATCH 2/3] Rework and simplify
---
llvm/include/llvm/Analysis/IVDescriptors.h | 50 ++++---
llvm/lib/Analysis/IVDescriptors.cpp | 127 +++++++-----------
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 31 +----
3 files changed, 79 insertions(+), 129 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index c644bac9a3ecc..a38f83d2e21ed 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -486,39 +486,49 @@ class InductionDescriptor {
/// A struct for saving information about monotonic variables.
/// Monotonic variable can be considered as a "conditional" induction variable:
/// its update happens only on loop iterations for which a certain predicate is
-/// satisfied. In this implementation the predicate is represented as an edge in
-/// loop CFG: variable is updated if this edge is executed on current loop
-/// iteration.
+/// satisfied.
class MonotonicDescriptor {
public:
- using Edge = std::pair<BasicBlock *, BasicBlock *>;
-
MonotonicDescriptor() = default;
- const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
- Instruction *getStepInst() const { return StepInst; }
- Edge getPredicateEdge() const { return PredEdge; }
- const SCEVAddRecExpr *getExpr() const { return Expr; }
+ MonotonicDescriptor(PHINode *HeaderPHI, PHINode *BackedgePHI,
+ Instruction *StepInst, const SCEVAddRecExpr *PhiSCEV)
+ : HeaderPHI(HeaderPHI), BackedgePHI(BackedgePHI), StepInst(StepInst),
+ PhiSCEV(PhiSCEV) {}
/// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
/// is monotonic, the monotonic descriptor \p D will contain the data
- /// describing this variable.
+ /// describing the PHI.
static bool isMonotonicPHI(PHINode *PN, const Loop *L,
MonotonicDescriptor &Desc, ScalarEvolution &SE);
- /// Returns true if \p Val is a monotonic variable in the loop \p L (in this
- /// case, the value should transitively contain monotonic phi as part of its
- /// calculation).
- static bool isMonotonicVal(Value *Val, const Loop *L,
- MonotonicDescriptor &Desc, ScalarEvolution &SE);
+ /// Returns the header PHI described by this descriptor.
+ PHINode *getHeaderPHI() const { return HeaderPHI; }
+
+ /// Returns the backedge PHI that selects between StepInst and the HeaderPHI.
+ PHINode *getBackedgePHI() const { return BackedgePHI; }
+
+ /// Returns the instruction that updates the value of the monotonic PHI.
+ Instruction *getStepInst() const { return StepInst; }
+
+ /// Returns the expression that represents the monotonic PHI. Note: The
+ /// conditional update is represented with a plain SCEVAddRec. This only holds
+ /// on iterations where the monotonic PHI is updated by StepInst.
+ const SCEVAddRecExpr *getPhiSCEV() const { return PhiSCEV; }
private:
- SmallPtrSet<PHINode *, 1> Chain;
- Instruction *StepInst;
- Edge PredEdge;
- const SCEVAddRecExpr *Expr;
+ /// The header PHI (this is the PHI described by the descriptor).
+ PHINode *HeaderPHI = nullptr;
+
+ /// The backedge PHI that selects between StepInst and the HeaderPHI.
+ PHINode *BackedgePHI = nullptr;
+
+ /// The instruction that updates the value of the monotonic PHI.
+ Instruction *StepInst = nullptr;
- bool setSCEV(const SCEV *NewExpr);
+ /// Expression that represents the monotonic PHI. Within the expression, the
+ /// conditional update is represented as an (unconditional) SCEVAddRec.
+ const SCEVAddRecExpr *PhiSCEV = nullptr;
};
} // end namespace llvm
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 595e8b6dc1f78..11d114da708da 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1697,83 +1697,63 @@ bool InductionDescriptor::isInductionPHI(
return true;
}
-bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
- auto *AddRec = dyn_cast<SCEVAddRecExpr>(NewExpr);
- if (!AddRec || !AddRec->isAffine())
- return false;
- Expr = AddRec;
- return true;
-}
-
// Recognize monotonic phi variable by matching the following pattern:
// loop_header:
-// %monotonic_phi = [%start, %preheader], [%chain_phi0, %latch]
+// %monotonic_phi = phi [ %start, %preheader ], [ %chain_phi0, %latch ]
+// br i1 %do_step, label %step_bb, label %latch
//
// step_bb:
// %step = add/gep %monotonic_phi, %step_val
-//
-// bbN:
-// %chain_phiN = [%monotonic_phi, ], [%step, ]
-//
-// ...
-//
-// bb1:
-// %chain_phi1 = [%monotonic_phi, ], [%chain_phi2, ]
+// br label %latch
//
// latch:
-// %chain_phi0 = [%monotonic_phi, %pred], [%chain_phi1, %pred]
+// %chain_phi0 = phi [ %monotonic_phi, %loop_header ], [ %step, %step_bb ]
+// br label %loop_header
//
-// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
-// and predicate is CFG edge %step_bb -> %bbN.
-bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
+// For this pattern, monotonic phi is described by {%start, +, %step}
+// recurrence and predicate is CFG edge %step_bb -> %latch.
+bool MonotonicDescriptor::isMonotonicPHI(PHINode *HeaderPHI, const Loop *L,
MonotonicDescriptor &Desc,
ScalarEvolution &SE) {
- if (!PN->getType()->isIntOrPtrTy() || PN->getParent() != L->getHeader())
+ if (!HeaderPHI->getType()->isIntOrPtrTy() ||
+ HeaderPHI->getParent() != L->getHeader())
return false;
- auto *BackEdgeInst =
- dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
- if (!BackEdgeInst)
+ auto *BackedgePHI =
+ dyn_cast<PHINode>(HeaderPHI->getIncomingValueForBlock(L->getLoopLatch()));
+ if (!BackedgePHI)
return false;
- PHINode *PHIChain = BackEdgeInst;
- std::optional<std::pair<Edge, Value *>> Inc;
- while (PHIChain) {
- Desc.Chain.insert(PHIChain);
- PHINode *NextPHIChain = nullptr;
- for (auto [Block, Incoming] :
- zip_equal(PHIChain->blocks(), PHIChain->incoming_values())) {
- if (Incoming == PN)
- continue;
- if (!Incoming->hasOneUse())
- return false;
- if (auto *IncomingPHI = dyn_cast<PHINode>(Incoming)) {
- if (NextPHIChain)
- return false;
- NextPHIChain = IncomingPHI;
- continue;
- }
- if (Inc || NextPHIChain)
- return false;
- Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
- }
- PHIChain = NextPHIChain;
- }
- if (!Inc)
+
+ // Ensure the only users of the backedge PHI are outside the loop or the
+ // header PHI.
+ bool BackedgeValueUsesValid = all_of(BackedgePHI->users(), [&](User *U) {
+ auto *UI = dyn_cast<Instruction>(U);
+ return UI == HeaderPHI || (UI && !L->contains(UI));
+ });
+ if (!BackedgeValueUsesValid)
return false;
- auto [PredEdge, StepOp] = *Inc;
- auto *StepInst = dyn_cast<Instruction>(StepOp);
+
+ // Find the step operation used to increment the value of the monotonic PHI.
+ // TODO: Support chains of PHIs.
+ Value *StepOp = nullptr;
+ for (Use &Incoming : BackedgePHI->incoming_values()) {
+ if (Incoming == HeaderPHI)
+ continue;
+ if (StepOp || !Incoming->hasOneUse())
+ return {};
+ StepOp = Incoming;
+ }
+
+ auto *StepInst = dyn_cast_if_present<Instruction>(StepOp);
if (!StepInst)
return false;
- Desc.StepInst = StepInst;
- Desc.PredEdge = PredEdge;
- // Construct SCEVAddRec for this value.
- Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
+ Value *Start = HeaderPHI->getIncomingValueForBlock(L->getLoopPreheader());
Value *Step = nullptr;
bool StepMatch =
- PN->getType()->isPointerTy()
- ? match(StepInst, m_PtrAdd(m_Specific(PN), m_Value(Step)))
- : match(StepInst, m_Add(m_Specific(PN), m_Value(Step)));
+ HeaderPHI->getType()->isPointerTy()
+ ? match(StepInst, m_PtrAdd(m_Specific(HeaderPHI), m_Value(Step)))
+ : match(StepInst, m_c_Add(m_Specific(HeaderPHI), m_Value(Step)));
if (!StepMatch || !L->isLoopInvariant(Step))
return false;
@@ -1790,30 +1770,15 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
}
- return Desc.setSCEV(
- SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags));
-}
-
-bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
- MonotonicDescriptor &Desc,
- ScalarEvolution &SE) {
- if (!Val->getType()->isIntOrPtrTy() || L->isLoopInvariant(Val))
+ const SCEV *PhiSCEV =
+ SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags);
+ const SCEVAddRecExpr *PhiAddRec = dyn_cast<SCEVAddRecExpr>(PhiSCEV);
+ if (!PhiAddRec || !PhiAddRec->isAffine())
return false;
- auto *CurInst = cast<Instruction>(Val);
-
- auto LoopVariantVal = [&](Value *V, bool AllowRepeats) {
- return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
- };
-
- while (!isa<PHINode>(CurInst)) {
- CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
- if (!CurInst)
- return false;
- };
- if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
- return false;
+ LLVM_DEBUG(dbgs() << "LV: Found a monotonic phi: HeaderPHI: " << *HeaderPHI
+ << ", StepInst: " << *StepInst << "\n");
- ValueToSCEVMapTy Map{{CurInst, Desc.getExpr()}};
- return Desc.setSCEV(SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
+ Desc = MonotonicDescriptor(HeaderPHI, BackedgePHI, StepInst, PhiAddRec);
+ return true;
}
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index 753abb0c7b93d..f346c43412f61 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -451,6 +451,7 @@ if.then:
%inc = add nsw i32 %monotonic, 1
%monotonic.prom = sext i32 %monotonic to i64
%arrayidx = getelementptr inbounds i32, ptr %dst, i64 %monotonic.prom
+ store i32 10, ptr %arrayidx, align 4
br label %for.inc
for.inc:
@@ -486,35 +487,15 @@ for.end:
assert(GEPInst->getName() == "arrayidx");
BasicBlock *IfEnd = &*(++FI);
assert(IfEnd->getName() == "for.inc");
- auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
- assert(ChainPhi->getName() == "monotonic.next");
// Check %monotonic descriptor.
MonotonicDescriptor Desc;
bool IsMonotonicPhi =
MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
EXPECT_TRUE(IsMonotonicPhi);
- auto &PhiChain = Desc.getChain();
- EXPECT_TRUE(PhiChain.size() == 1 && PhiChain.contains(ChainPhi));
EXPECT_EQ(Desc.getStepInst(), StepInst);
- EXPECT_EQ(Desc.getPredicateEdge(),
- MonotonicDescriptor::Edge(IfThen, IfEnd));
auto *StartSCEV = SE.getConstant(Phi->getType(), 0);
auto *StepSCEV = SE.getConstant(Phi->getType(), 1);
- EXPECT_EQ(Desc.getExpr(),
- SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
- // Check %arrayidx descriptor.
- bool IsMonotonicVal =
- MonotonicDescriptor::isMonotonicVal(GEPInst, L, Desc, SE);
- EXPECT_TRUE(IsMonotonicVal);
- // Chain, StepInst and PredicateEdge are the same with %monotonic.
- auto &ValChain = Desc.getChain();
- EXPECT_TRUE(ValChain.size() == 1 && ValChain.contains(ChainPhi));
- EXPECT_EQ(Desc.getStepInst(), StepInst);
- EXPECT_EQ(Desc.getPredicateEdge(),
- MonotonicDescriptor::Edge(IfThen, IfEnd));
- StartSCEV = SE.getSCEV(F.getArg(0));
- StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
- EXPECT_EQ(Desc.getExpr(),
+ EXPECT_EQ(Desc.getPhiSCEV(),
SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
});
}
@@ -566,20 +547,14 @@ for.end:
assert(StepInst->getName() == "inc");
BasicBlock *IfEnd = &*(++FI);
assert(IfEnd->getName() == "for.inc");
- auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
- assert(ChainPhi->getName() == "monotonic.next");
MonotonicDescriptor Desc;
bool IsMonotonicPhi =
MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
EXPECT_TRUE(IsMonotonicPhi);
- auto &Chain = Desc.getChain();
- EXPECT_TRUE(Chain.size() == 1 && Chain.contains(ChainPhi));
EXPECT_EQ(Desc.getStepInst(), StepInst);
- EXPECT_EQ(Desc.getPredicateEdge(),
- MonotonicDescriptor::Edge(IfThen, IfEnd));
auto *StartSCEV = SE.getSCEV(F.getArg(0));
auto *StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
- EXPECT_EQ(Desc.getExpr(),
+ EXPECT_EQ(Desc.getPhiSCEV(),
SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
});
}
>From 10c9b61c760ff5cd4a775664b0ee21b2066abcda Mon Sep 17 00:00:00 2001
From: Benjamin Maxwell <benjamin.maxwell at arm.com>
Date: Fri, 21 Aug 2026 15:23:41 +0000
Subject: [PATCH 3/3] Fixups
---
llvm/include/llvm/Analysis/IVDescriptors.h | 2 +-
llvm/lib/Analysis/IVDescriptors.cpp | 19 +++---
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 67 ++++++++++++++++---
3 files changed, 69 insertions(+), 19 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index a38f83d2e21ed..9da788f263e64 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -486,7 +486,7 @@ class InductionDescriptor {
/// A struct for saving information about monotonic variables.
/// Monotonic variable can be considered as a "conditional" induction variable:
/// its update happens only on loop iterations for which a certain predicate is
-/// satisfied.
+/// satisfied. The step of the monotonic variable must be loop-invariant.
class MonotonicDescriptor {
public:
MonotonicDescriptor() = default;
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 11d114da708da..8e6f72cf2cda9 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1715,7 +1715,7 @@ bool InductionDescriptor::isInductionPHI(
bool MonotonicDescriptor::isMonotonicPHI(PHINode *HeaderPHI, const Loop *L,
MonotonicDescriptor &Desc,
ScalarEvolution &SE) {
- if (!HeaderPHI->getType()->isIntOrPtrTy() ||
+ if (!L->getLoopPreheader() || !HeaderPHI->getType()->isIntOrPtrTy() ||
HeaderPHI->getParent() != L->getHeader())
return false;
auto *BackedgePHI =
@@ -1734,16 +1734,15 @@ bool MonotonicDescriptor::isMonotonicPHI(PHINode *HeaderPHI, const Loop *L,
// Find the step operation used to increment the value of the monotonic PHI.
// TODO: Support chains of PHIs.
- Value *StepOp = nullptr;
- for (Use &Incoming : BackedgePHI->incoming_values()) {
- if (Incoming == HeaderPHI)
- continue;
- if (StepOp || !Incoming->hasOneUse())
- return {};
- StepOp = Incoming;
- }
+ Value *StepOp = find_singleton<Value>(
+ BackedgePHI->incoming_values(),
+ [&](Use &Incoming, bool /*AllowRepeasts*/) {
+ return Incoming != HeaderPHI ? Incoming.get() : nullptr;
+ });
+ if (!StepOp || !StepOp->hasOneUse())
+ return false;
- auto *StepInst = dyn_cast_if_present<Instruction>(StepOp);
+ auto *StepInst = dyn_cast<Instruction>(StepOp);
if (!StepInst)
return false;
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index f346c43412f61..454cbd23b4210 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -481,12 +481,6 @@ for.end:
BBI = IfThen->begin();
Instruction *StepInst = &*BBI;
assert(StepInst->getName() == "inc");
- Instruction *ExtInst = &*(++BBI);
- assert(ExtInst->getName() == "monotonic.prom");
- Instruction *GEPInst = &*(++BBI);
- assert(GEPInst->getName() == "arrayidx");
- BasicBlock *IfEnd = &*(++FI);
- assert(IfEnd->getName() == "for.inc");
// Check %monotonic descriptor.
MonotonicDescriptor Desc;
bool IsMonotonicPhi =
@@ -545,8 +539,6 @@ for.end:
assert(IfThen->getName() == "if.then");
Instruction *StepInst = &*(IfThen->begin());
assert(StepInst->getName() == "inc");
- BasicBlock *IfEnd = &*(++FI);
- assert(IfEnd->getName() == "for.inc");
MonotonicDescriptor Desc;
bool IsMonotonicPhi =
MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
@@ -558,3 +550,62 @@ for.end:
SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
});
}
+
+TEST(IVDescriptorsTest, InvalidMonotonicExtraStep) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %dst, i1 %cond, i1 %cond2, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi i32 [ 0, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = add nsw i32 %monotonic, 1
+ %monotonic.prom = sext i32 %monotonic to i64
+ %arrayidx = getelementptr inbounds i32, ptr %dst, i64 %monotonic.prom
+ store i32 10, ptr %arrayidx, align 4
+ br i1 %cond2, label %if.then1, label %for.inc
+if.then1:
+ %inc2 = add nsw i32 %monotonic, 2
+ br label %for.inc
+for.inc:
+ %monotonic.next = phi i32 [ %inc, %if.then ], [ %inc2, %if.then1 ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ BBI = IfThen->begin();
+ Instruction *StepInst = &*BBI;
+ assert(StepInst->getName() == "inc");
+ // Check %monotonic descriptor.
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_FALSE(IsMonotonicPhi);
+ });
+}
More information about the llvm-commits
mailing list