[llvm] [IVDescriptors] Implement MonotonicDescriptor (PR #214490)
Benjamin Maxwell via llvm-commits
llvm-commits at lists.llvm.org
Thu Aug 6 06:34:43 PDT 2026
https://github.com/MacDue created https://github.com/llvm/llvm-project/pull/214490
RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442
"Monotonic" variable is similar to induction variable, but its value is updated under some condition, e.g.:
```
int idx = 0;
for(int i = 0; i < n; ++i) {
// some uses of idx
if (cond)
++idx;
}
```
In this example, `i` is induction variable and `idx` is monotonic variable: it's updated only when cond == true. In LLVM IR, this looks like:
```
loop_header:
%monotonic_phi = [%start, %prehader], [ %chain_phi0, %latch]
step_bb:
%step = add/gep %monotonic_phi, %step_val
bbN:
%chain_phiN = [%step, %step_bb], [%monotonic_phi, %some_pred]
...
bb1:
%chain_phi1 = [%monotonic_phi, %some_pred], [%chain_phi2, %some_pred]
loop_latch:
%chain_phi0 = [%monotonic_phi, %some_pred], [%chain_phi1, %some_pred]
```
We start the analysis from the header phi (%monotonic_phi). Its backedge value (%chain_phi0) can be either unmodified (%monotonic_val) or the incremented one (%step) depending on some condition. Therefore, all incoming values in chain phi should be %monotonic_phi except the one that points to the next chain phi or the step instruction. The analysis stops when the correct step instruction is found (correct step instruction is in form add/gep %monotonic_phi, %step_val, and %step_val is loop-invariant). In this way, %monotonic_val is incremented on %step_val only when last chain phi "selects" %step incoming value; in other words, when step_bb -> bbN edge is executed. The condition of monotonic variable update is stored as such CFG edge (`getPredicateEdge()` method). The other MonotonicDescriptor methods are:
1. `getChain()` - returns set of chain phis (%chain_phi0, ..., %chain_phiN)
2. `getStepInst()` - returns %step instruction (either add or getlementptr)
3. `getExpr()` - returns SCEVAddRec in assumption that condition of update is always true (SCEVAddRec + condtion predicate fully describes evolution of monotonic variable in the loop)
MonotonicDescriptor also implements `isMonotonicVal` method. This routine will allow to recognize values which have monotonic phi as a transitive dependency, e.g. array subscript operators `arr[idx]`, where idx is monotonic phi. The descriptor of monotonic value is equivalent to the descriptor of monotonic phi, except its SCEV: we obtain the SCEV of monotonic value and replace the SCEVUnknown expression that corresponds to monotonic phi with the SCEVAddRec from monotonic phi descriptor. The resulted SCEV shows the evolution of monotonic value in loop iterations when predicate is true (similarly to SCEV).
*Restrictions*
1. We only support "post-increment" update of monotonic vars; in other words, uses of chain phis (other than in other chain phis) are prohibited. It's only alowed to use %monotonic_phi outside of described chain pattern.
2. We don't support `select` instructions in monotonic patterns (instead of chain phis). In theory this can be implemented, but on practice we want to use MonotonicDescriptor to recognize expandloads/compressstores, where step instruction is placed near memory instruction, so if-conversion to select is not applied there (because basic block can't be eliminated anyway). So, this restriction doesn't limit our abilities to recognize expandloads/compressstore patterns.
This is a continuation Sergey Kachkov's patch (#140720).
---
<sub>Stack created with <a href="https://github.com/github/gh-stack">GitHub Stacks CLI</a> • <a href="https://gh.io/stacks-feedback">Give Feedback 💬</a></sub>
>From 6398d5236dc87790598ef0104320662be7ec71ef Mon Sep 17 00:00:00 2001
From: Sergey Kachkov <sergey.kachkov at syntacore.com>
Date: Wed, 29 Jan 2025 15:27:01 +0300
Subject: [PATCH] [IVDescriptors] Implement MonotonicDescriptor
RFC link: https://discourse.llvm.org/t/rfc-loop-vectorization-of-compress-store-expand-load-patterns/86442
"Monotonic" variable is similar to induction variable, but its value is updated under some condition, e.g.:
```
int idx = 0;
for(int i = 0; i < n; ++i) {
// some uses of idx
if (cond)
++idx;
}
```
In this example, `i` is induction variable and `idx` is monotonic variable: it's updated only when cond == true. In LLVM IR, this looks like:
```
loop_header:
%monotonic_phi = [%start, %prehader], [ %chain_phi0, %latch]
step_bb:
%step = add/gep %monotonic_phi, %step_val
bbN:
%chain_phiN = [%step, %step_bb], [%monotonic_phi, %some_pred]
...
bb1:
%chain_phi1 = [%monotonic_phi, %some_pred], [%chain_phi2, %some_pred]
loop_latch:
%chain_phi0 = [%monotonic_phi, %some_pred], [%chain_phi1, %some_pred]
```
We start the analysis from the header phi (%monotonic_phi). Its backedge value (%chain_phi0) can be either unmodified (%monotonic_val) or the incremented one (%step) depending on some condition. Therefore, all incoming values in chain phi should be %monotonic_phi except the one that points to the next chain phi or the step instruction. The analysis stops when the correct step instruction is found (correct step instruction is in form add/gep %monotonic_phi, %step_val, and %step_val is loop-invariant). In this way, %monotonic_val is incremented on %step_val only when last chain phi "selects" %step incoming value; in other words, when step_bb -> bbN edge is executed. The condition of monotonic variable update is stored as such CFG edge (`getPredicateEdge()` method). The other MonotonicDescriptor methods are:
1. `getChain()` - returns set of chain phis (%chain_phi0, ..., %chain_phiN)
2. `getStepInst()` - returns %step instruction (either add or getlementptr)
3. `getExpr()` - returns SCEVAddRec in assumption that condition of update is always true (SCEVAddRec + condtion predicate fully describes evolution of monotonic variable in the loop)
MonotonicDescriptor also implements `isMonotonicVal` method. This routine will allow to recognize values which have monotonic phi as a transitive dependency, e.g. array subscript operators `arr[idx]`, where idx is monotonic phi. The descriptor of monotonic value is equivalent to the descriptor of monotonic phi, except its SCEV: we obtain the SCEV of monotonic value and replace the SCEVUnknown expression that corresponds to monotonic phi with the SCEVAddRec from monotonic phi descriptor. The resulted SCEV shows the evolution of monotonic value in loop iterations when predicate is true (similarly to SCEV).
*Restrictions*
1. We only support "post-increment" update of monotonic vars; in other words, uses of chain phis (other than in other chain phis) are prohibited. It's only alowed to use %monotonic_phi outside of described chain pattern.
2. We don't support `select` instructions in monotonic patterns (instead of chain phis). In theory this can be implemented, but on practice we want to use MonotonicDescriptor to recognize expandloads/compressstores, where step instruction is placed near memory instruction, so if-conversion to select is not applied there (because basic block can't be eliminated anyway). So, this restriction doesn't limit our abilities to recognize expandloads/compressstore patterns.
This is a continuation Sergey Kachkov's patch (#140720).
---
llvm/include/llvm/Analysis/IVDescriptors.h | 39 +++
llvm/lib/Analysis/IVDescriptors.cpp | 121 +++++++++
llvm/unittests/Analysis/IVDescriptorsTest.cpp | 243 ++++++++++++++----
3 files changed, 358 insertions(+), 45 deletions(-)
diff --git a/llvm/include/llvm/Analysis/IVDescriptors.h b/llvm/include/llvm/Analysis/IVDescriptors.h
index bad372421dfae..c644bac9a3ecc 100644
--- a/llvm/include/llvm/Analysis/IVDescriptors.h
+++ b/llvm/include/llvm/Analysis/IVDescriptors.h
@@ -29,6 +29,7 @@ class PredicatedScalarEvolution;
class ScalarEvolution;
class SCEV;
class SCEVPredicate;
+class SCEVAddRecExpr;
class StoreInst;
/// These are the kinds of recurrences that we support.
@@ -482,6 +483,44 @@ class InductionDescriptor {
SmallVector<const SCEVPredicate *, 2> NoWrapPredicates;
};
+/// A struct for saving information about monotonic variables.
+/// Monotonic variable can be considered as a "conditional" induction variable:
+/// its update happens only on loop iterations for which a certain predicate is
+/// satisfied. In this implementation the predicate is represented as an edge in
+/// loop CFG: variable is updated if this edge is executed on current loop
+/// iteration.
+class MonotonicDescriptor {
+public:
+ using Edge = std::pair<BasicBlock *, BasicBlock *>;
+
+ MonotonicDescriptor() = default;
+
+ const SmallPtrSetImpl<PHINode *> &getChain() const { return Chain; }
+ Instruction *getStepInst() const { return StepInst; }
+ Edge getPredicateEdge() const { return PredEdge; }
+ const SCEVAddRecExpr *getExpr() const { return Expr; }
+
+ /// Returns true if \p PN is a monotonic variable in the loop \p L. If \p PN
+ /// is monotonic, the monotonic descriptor \p D will contain the data
+ /// describing this variable.
+ static bool isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+ /// Returns true if \p Val is a monotonic variable in the loop \p L (in this
+ /// case, the value should transitively contain monotonic phi as part of its
+ /// calculation).
+ static bool isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc, ScalarEvolution &SE);
+
+private:
+ SmallPtrSet<PHINode *, 1> Chain;
+ Instruction *StepInst;
+ Edge PredEdge;
+ const SCEVAddRecExpr *Expr;
+
+ bool setSCEV(const SCEV *NewExpr);
+};
+
} // end namespace llvm
#endif // LLVM_ANALYSIS_IVDESCRIPTORS_H
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 800b64e9a29af..595e8b6dc1f78 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -1696,3 +1696,124 @@ bool InductionDescriptor::isInductionPHI(
/*InductionBinOp=*/nullptr, /*Casts=*/nullptr, Preds);
return true;
}
+
+bool MonotonicDescriptor::setSCEV(const SCEV *NewExpr) {
+ auto *AddRec = dyn_cast<SCEVAddRecExpr>(NewExpr);
+ if (!AddRec || !AddRec->isAffine())
+ return false;
+ Expr = AddRec;
+ return true;
+}
+
+// Recognize monotonic phi variable by matching the following pattern:
+// loop_header:
+// %monotonic_phi = [%start, %preheader], [%chain_phi0, %latch]
+//
+// step_bb:
+// %step = add/gep %monotonic_phi, %step_val
+//
+// bbN:
+// %chain_phiN = [%monotonic_phi, ], [%step, ]
+//
+// ...
+//
+// bb1:
+// %chain_phi1 = [%monotonic_phi, ], [%chain_phi2, ]
+//
+// latch:
+// %chain_phi0 = [%monotonic_phi, %pred], [%chain_phi1, %pred]
+//
+// For this pattern, monotonic phi is described by {%start, +, %step} recurrence
+// and predicate is CFG edge %step_bb -> %bbN.
+bool MonotonicDescriptor::isMonotonicPHI(PHINode *PN, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!PN->getType()->isIntOrPtrTy() || PN->getParent() != L->getHeader())
+ return false;
+ auto *BackEdgeInst =
+ dyn_cast<PHINode>(PN->getIncomingValueForBlock(L->getLoopLatch()));
+ if (!BackEdgeInst)
+ return false;
+ PHINode *PHIChain = BackEdgeInst;
+ std::optional<std::pair<Edge, Value *>> Inc;
+ while (PHIChain) {
+ Desc.Chain.insert(PHIChain);
+ PHINode *NextPHIChain = nullptr;
+ for (auto [Block, Incoming] :
+ zip_equal(PHIChain->blocks(), PHIChain->incoming_values())) {
+ if (Incoming == PN)
+ continue;
+ if (!Incoming->hasOneUse())
+ return false;
+ if (auto *IncomingPHI = dyn_cast<PHINode>(Incoming)) {
+ if (NextPHIChain)
+ return false;
+ NextPHIChain = IncomingPHI;
+ continue;
+ }
+ if (Inc || NextPHIChain)
+ return false;
+ Inc = std::make_pair(Edge{Block, PHIChain->getParent()}, Incoming.get());
+ }
+ PHIChain = NextPHIChain;
+ }
+ if (!Inc)
+ return false;
+ auto [PredEdge, StepOp] = *Inc;
+ auto *StepInst = dyn_cast<Instruction>(StepOp);
+ if (!StepInst)
+ return false;
+ Desc.StepInst = StepInst;
+ Desc.PredEdge = PredEdge;
+
+ // Construct SCEVAddRec for this value.
+ Value *Start = PN->getIncomingValueForBlock(L->getLoopPreheader());
+
+ Value *Step = nullptr;
+ bool StepMatch =
+ PN->getType()->isPointerTy()
+ ? match(StepInst, m_PtrAdd(m_Specific(PN), m_Value(Step)))
+ : match(StepInst, m_Add(m_Specific(PN), m_Value(Step)));
+ if (!StepMatch || !L->isLoopInvariant(Step))
+ return false;
+
+ SCEV::NoWrapFlags WrapFlags = SCEV::FlagAnyWrap;
+ if (auto *GEP = dyn_cast<GEPOperator>(StepInst)) {
+ if (GEP->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (GEP->hasNoUnsignedSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ } else if (auto *OBO = dyn_cast<OverflowingBinaryOperator>(StepInst)) {
+ if (OBO->hasNoUnsignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNUW);
+ if (OBO->hasNoSignedWrap())
+ WrapFlags = ScalarEvolution::setFlags(WrapFlags, SCEV::FlagNSW);
+ }
+
+ return Desc.setSCEV(
+ SE.getAddRecExpr(SE.getSCEV(Start), SE.getSCEV(Step), L, WrapFlags));
+}
+
+bool MonotonicDescriptor::isMonotonicVal(Value *Val, const Loop *L,
+ MonotonicDescriptor &Desc,
+ ScalarEvolution &SE) {
+ if (!Val->getType()->isIntOrPtrTy() || L->isLoopInvariant(Val))
+ return false;
+ auto *CurInst = cast<Instruction>(Val);
+
+ auto LoopVariantVal = [&](Value *V, bool AllowRepeats) {
+ return L->isLoopInvariant(V) ? nullptr : cast<Instruction>(V);
+ };
+
+ while (!isa<PHINode>(CurInst)) {
+ CurInst = find_singleton<Instruction>(CurInst->operands(), LoopVariantVal);
+ if (!CurInst)
+ return false;
+ };
+
+ if (!isMonotonicPHI(cast<PHINode>(CurInst), L, Desc, SE))
+ return false;
+
+ ValueToSCEVMapTy Map{{CurInst, Desc.getExpr()}};
+ return Desc.setSCEV(SCEVParameterRewriter::rewrite(SE.getSCEV(Val), SE, Map));
+}
diff --git a/llvm/unittests/Analysis/IVDescriptorsTest.cpp b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
index faf30fd322c10..09f52cec62c34 100644
--- a/llvm/unittests/Analysis/IVDescriptorsTest.cpp
+++ b/llvm/unittests/Analysis/IVDescriptorsTest.cpp
@@ -10,6 +10,7 @@
#include "llvm/Analysis/AssumptionCache.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/ScalarEvolution.h"
+#include "llvm/Analysis/ScalarEvolutionExpressions.h"
#include "llvm/Analysis/TargetLibraryInfo.h"
#include "llvm/AsmParser/Parser.h"
#include "llvm/IR/Dominators.h"
@@ -339,6 +340,203 @@ TEST(IVDescriptorsTest, UnsupportedFindLastPhi) {
});
}
+// Make sure isReductionPHI doesn't crash when SE is not passed to it.
+TEST(IVDescriptorsTest, InvariantStoreNoSCEV) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M = parseIR(Context, R"(
+ define void @smax_with_invariant_store_user(ptr noalias %src, ptr %dst, i64 %n) {
+ entry:
+ br label %loop
+
+ loop:
+ %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
+ %max = phi i32 [ 0, %entry ], [ %max.next, %loop ]
+ %gep.src = getelementptr inbounds i32, ptr %src, i64 %iv
+ %l = load i32, ptr %gep.src, align 4
+ %max.next = tail call i32 @llvm.smax.i32(i32 %max, i32 %l)
+ store i32 %max.next, ptr %dst, align 4
+ %iv.next = add i64 %iv, 1
+ %ec = icmp eq i64 %iv, %n
+ br i1 %ec, label %exit, label %loop
+
+ exit:
+ ret void
+ })");
+
+ runWithLoopInfoAndSE(*M, "smax_with_invariant_store_user",
+ [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "loop");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ ++BBI;
+ PHINode *Phi = dyn_cast<PHINode>(&*BBI);
+ EXPECT_NE(Phi, nullptr);
+ EXPECT_EQ(Phi->getName(), "max");
+ RecurrenceDescriptor Rdx;
+ bool IsRdxPhi =
+ RecurrenceDescriptor::isReductionPHI(Phi, L, Rdx);
+ EXPECT_FALSE(IsRdxPhi);
+ });
+}
+
+TEST(IVDescriptorsTest, MonotonicIntVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %dst, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi i32 [ 0, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = add nsw i32 %monotonic, 1
+ %monotonic.prom = sext i32 %monotonic to i64
+ %arrayidx = getelementptr inbounds i32, ptr %dst, i64 %monotonic.prom
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi i32 [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ BBI = IfThen->begin();
+ Instruction *StepInst = &*BBI;
+ assert(StepInst->getName() == "inc");
+ Instruction *ExtInst = &*(++BBI);
+ assert(ExtInst->getName() == "monotonic.prom");
+ Instruction *GEPInst = &*(++BBI);
+ assert(GEPInst->getName() == "arrayidx");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ // Check %monotonic descriptor.
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &PhiChain = Desc.getChain();
+ EXPECT_TRUE(PhiChain.size() == 1 && PhiChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getConstant(Phi->getType(), 0);
+ auto *StepSCEV = SE.getConstant(Phi->getType(), 1);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ // Check %arrayidx descriptor.
+ bool IsMonotonicVal =
+ MonotonicDescriptor::isMonotonicVal(GEPInst, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicVal);
+ // Chain, StepInst and PredicateEdge are the same with %monotonic.
+ auto &ValChain = Desc.getChain();
+ EXPECT_TRUE(ValChain.size() == 1 && ValChain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ StartSCEV = SE.getSCEV(F.getArg(0));
+ StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
+
+TEST(IVDescriptorsTest, MonotonicPtrVar) {
+ // Parse the module.
+ LLVMContext Context;
+
+ std::unique_ptr<Module> M =
+ parseIR(Context,
+ R"(define void @foo(ptr %start, i1 %cond, i64 %n) {
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %monotonic = phi ptr [ %start, %entry ], [ %monotonic.next, %for.inc ]
+ br i1 %cond, label %if.then, label %for.inc
+
+if.then:
+ %inc = getelementptr inbounds i8, ptr %monotonic, i64 4
+ br label %for.inc
+
+for.inc:
+ %monotonic.next = phi ptr [ %inc, %if.then ], [ %monotonic, %for.body ]
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond.not = icmp eq i64 %i.next, %n
+ br i1 %exitcond.not, label %for.end, label %for.body
+
+for.end:
+ ret void
+})");
+
+ runWithLoopInfoAndSE(
+ *M, "foo", [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
+ Function::iterator FI = F.begin();
+ // First basic block is entry - skip it.
+ BasicBlock *Header = &*(++FI);
+ assert(Header->getName() == "for.body");
+ Loop *L = LI.getLoopFor(Header);
+ EXPECT_NE(L, nullptr);
+ BasicBlock::iterator BBI = Header->begin();
+ assert((&*BBI)->getName() == "i");
+ PHINode *Phi = dyn_cast<PHINode>(&*(++BBI));
+ assert(Phi->getName() == "monotonic");
+ BasicBlock *IfThen = &*(++FI);
+ assert(IfThen->getName() == "if.then");
+ Instruction *StepInst = &*(IfThen->begin());
+ assert(StepInst->getName() == "inc");
+ BasicBlock *IfEnd = &*(++FI);
+ assert(IfEnd->getName() == "for.inc");
+ auto *ChainPhi = dyn_cast<PHINode>(&*(IfEnd->begin()));
+ assert(ChainPhi->getName() == "monotonic.next");
+ MonotonicDescriptor Desc;
+ bool IsMonotonicPhi =
+ MonotonicDescriptor::isMonotonicPHI(Phi, L, Desc, SE);
+ EXPECT_TRUE(IsMonotonicPhi);
+ auto &Chain = Desc.getChain();
+ EXPECT_TRUE(Chain.size() == 1 && Chain.contains(ChainPhi));
+ EXPECT_EQ(Desc.getStepInst(), StepInst);
+ EXPECT_EQ(Desc.getPredicateEdge(),
+ MonotonicDescriptor::Edge(IfThen, IfEnd));
+ auto *StartSCEV = SE.getSCEV(F.getArg(0));
+ auto *StepSCEV = SE.getConstant(StartSCEV->getType(), 4);
+ EXPECT_EQ(Desc.getExpr(),
+ SE.getAddRecExpr(StartSCEV, StepSCEV, L, SCEV::FlagNW));
+ });
+}
+
// This tests that a min/max recurrence with a vector-typed phi is recognized.
TEST(IVDescriptorsTest, VectorSMaxRednIdentity) {
// Parse the module.
@@ -385,48 +583,3 @@ for.end:
EXPECT_EQ(Kind, RecurKind::SMax);
});
}
-
-// Make sure isReductionPHI doesn't crash when SE is not passed to it.
-TEST(IVDescriptorsTest, InvariantStoreNoSCEV) {
- // Parse the module.
- LLVMContext Context;
-
- std::unique_ptr<Module> M = parseIR(Context, R"(
- define void @smax_with_invariant_store_user(ptr noalias %src, ptr %dst, i64 %n) {
- entry:
- br label %loop
-
- loop:
- %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ]
- %max = phi i32 [ 0, %entry ], [ %max.next, %loop ]
- %gep.src = getelementptr inbounds i32, ptr %src, i64 %iv
- %l = load i32, ptr %gep.src, align 4
- %max.next = tail call i32 @llvm.smax.i32(i32 %max, i32 %l)
- store i32 %max.next, ptr %dst, align 4
- %iv.next = add i64 %iv, 1
- %ec = icmp eq i64 %iv, %n
- br i1 %ec, label %exit, label %loop
-
- exit:
- ret void
- })");
-
- runWithLoopInfoAndSE(*M, "smax_with_invariant_store_user",
- [&](Function &F, LoopInfo &LI, ScalarEvolution &SE) {
- Function::iterator FI = F.begin();
- // First basic block is entry - skip it.
- BasicBlock *Header = &*(++FI);
- assert(Header->getName() == "loop");
- Loop *L = LI.getLoopFor(Header);
- EXPECT_NE(L, nullptr);
- BasicBlock::iterator BBI = Header->begin();
- ++BBI;
- PHINode *Phi = dyn_cast<PHINode>(&*BBI);
- EXPECT_NE(Phi, nullptr);
- EXPECT_EQ(Phi->getName(), "max");
- RecurrenceDescriptor Rdx;
- bool IsRdxPhi =
- RecurrenceDescriptor::isReductionPHI(Phi, L, Rdx);
- EXPECT_FALSE(IsRdxPhi);
- });
-}
More information about the llvm-commits
mailing list