[llvm-branch-commits] [llvm] [LV] Vectorize down-counting floating-point argmin/argmax reductions (PR #209827)
via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Sun Jul 19 23:35:25 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-llvm-analysis
Author: Madhur Amilkanthwar (madhur13490)
<details>
<summary>Changes</summary>
Extend the multi-use min/max reduction coupling so a floating-point
min/max value reduction can be paired with a down-counting (FindFirst)
index reduction, enabling argmin/argmax vectorization for loops whose
induction counts down.
Analysis (IVDescriptors):
- Recognize select-based FP min/max recurrences, not just the
intrinsic form.
- For the shared-compare argmin/argmax shape (the reduction compare
also feeds an index select), take the required NaN-free and
signed-zero-free facts from the compare when the select itself does
not carry them. Plain min/max reductions keep the strict
flags-on-the-select rule.
VPlan (handleMultiUseReductions):
- Accept a select as the value reduction's min/max operation and read
its value operands accordingly.
- Tolerate the shared compare feeding both the value select and the
index select, skipping the value select when locating the index
select.
- Support the down-counting FindFirst index reduction (UMin/SMin) for
floating-point, reconstructing the stored index from the base
induction plus an additive offset (e.g. iv-1).
- Fall back to scalar safely for shapes that are not handled.
Scope is intentionally limited to keep the change small:
- TODO: floating-point up-counting (FindLast) argmin/argmax.
- TODO: integer down-counting (FindFirst) argmin/argmax.
Add lit tests covering the down-counting floating-point argmin,
including the VF=1 scalar-plan path.
---
Patch is 34.55 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/209827.diff
5 Files Affected:
- (modified) llvm/lib/Analysis/IVDescriptors.cpp (+29-5)
- (modified) llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp (+221-43)
- (modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+1)
- (modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp (+26-2)
- (added) llvm/test/Transforms/LoopVectorize/AArch64/select-index-decreasing.ll (+127)
``````````diff
diff --git a/llvm/lib/Analysis/IVDescriptors.cpp b/llvm/lib/Analysis/IVDescriptors.cpp
index 9d30928d1751c..c9cb3fe6fddd0 100644
--- a/llvm/lib/Analysis/IVDescriptors.cpp
+++ b/llvm/lib/Analysis/IVDescriptors.cpp
@@ -319,15 +319,33 @@ static RecurrenceDescriptor getMinMaxRecurrence(PHINode *Phi, Loop *TheLoop,
return {};
RK = CurRK;
- // Check required fast-math flags for FP recurrences.
+ // A select-based min/max whose compare also drives another select is the
+ // argmin/argmax shape: one select carries the value, the other the index.
+ // Detect it here so the fast-math check and chain handling treat it as a
+ // multi-use reduction rather than a plain one.
+ auto *SI = dyn_cast<SelectInst>(Cur);
+ bool FeedsOtherSelect =
+ SI && any_of(SI->getCondition()->users(),
+ [&](User *U) { return U != SI && isa<SelectInst>(U); });
+
+ // Check required fast-math flags for FP recurrences. A plain min/max
+ // reduction must carry the flags on the select itself. Only for the
+ // shared-compare argmin/argmax shape do the NaN-free/signed-zero-free facts
+ // legitimately live on the compare, so fall back to the condition's flags
+ // there.
if (RecurrenceDescriptor::isFPMinMaxRecurrenceKind(CurRK)) {
auto CurFMF = hasRequiredFastMathFlags(cast<FPMathOperator>(Cur), RK);
+ if (!CurFMF && FeedsOtherSelect)
+ if (auto *FC = dyn_cast<FCmpInst>(SI->getCondition()))
+ CurFMF = hasRequiredFastMathFlags(cast<FPMathOperator>(FC), RK);
if (!CurFMF)
return {};
FMF &= *CurFMF;
}
- if (auto *SI = dyn_cast<SelectInst>(I))
+ // Keep the compare external only for the argmin/argmax shape, so the
+ // multi-use reduction path below can recognize it.
+ if (SI && !FeedsOtherSelect)
Chain.insert(SI->getCondition());
if (A == Phi || B == Phi)
@@ -355,12 +373,18 @@ static RecurrenceDescriptor getMinMaxRecurrence(PHINode *Phi, Loop *TheLoop,
GetMinMaxRK(U, A, B) == RecurKind::None;
});
if (PhiHasInvalidUses) {
- if (!RecurrenceDescriptor::isIntMinMaxRecurrenceKind(RK) ||
- !BackedgeValue->hasOneUse())
+ // Accept integer (llvm.smin/smax intrinsic) and floating-point
+ // (select-based) min/max value reductions with a single-use backedge value.
+ // NaN-free semantics are required to reorder the parallel FP reduction.
+ bool IsIntArgmin = RecurrenceDescriptor::isIntMinMaxRecurrenceKind(RK);
+ bool IsFPArgmin =
+ RecurrenceDescriptor::isFPMinMaxRecurrenceKind(RK) && FMF.noNaNs();
+ if ((!IsIntArgmin && !IsFPArgmin) || !BackedgeValue->hasOneUse())
return {};
return RecurrenceDescriptor(
Phi->getIncomingValueForBlock(TheLoop->getLoopPreheader()),
- /*Exit=*/nullptr, /*Store=*/nullptr, RK, FastMathFlags(),
+ /*Exit=*/nullptr, /*Store=*/nullptr, RK,
+ IsIntArgmin ? FastMathFlags() : FMF,
/*ExactFP=*/nullptr, Phi->getType(), /*IsMultiUse=*/true);
}
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 1a135b8549514..b14cf058eac23 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -1626,6 +1626,36 @@ static VPInstruction *findFindIVSelect(VPValue *BackedgeVal) {
}));
}
+/// Recover the wide induction underlying the value \p IVOp that a FindIV select
+/// stores. \p IVOp is either the induction itself (then \p Offset is left null)
+/// or an affine increment add(induction, loop-invariant), such as the iv-1
+/// stored by a down-counting argmin loop (then \p Offset is set to the value to
+/// add to the reconstructed index). Returns the base induction, or nullptr if
+/// \p IVOp is not a (possibly offset) wide induction.
+static VPWidenIntOrFpInductionRecipe *getFindIVBaseInduction(VPValue *IVOp,
+ VPValue *&Offset) {
+ Offset = nullptr;
+ // A narrowed store keeps the wide IV; any offset is reapplied after the
+ // index is reconstructed.
+ match(IVOp, m_TruncOrSelf(m_VPValue(IVOp)));
+ if (auto *WideIV = dyn_cast<VPWidenIntOrFpInductionRecipe>(IVOp))
+ return WideIV;
+
+ // Otherwise accept add(IV, loop-invariant), e.g. iv-1 as add(iv, -1).
+ VPValue *LHS, *RHS;
+ if (!match(IVOp, m_Add(m_VPValue(LHS), m_VPValue(RHS))))
+ return nullptr;
+ auto *WideIV = dyn_cast<VPWidenIntOrFpInductionRecipe>(LHS);
+ if (!WideIV) {
+ std::swap(LHS, RHS);
+ WideIV = dyn_cast<VPWidenIntOrFpInductionRecipe>(LHS);
+ }
+ if (!WideIV || !RHS->isDefinedOutsideLoopRegions())
+ return nullptr;
+ Offset = RHS;
+ return WideIV;
+}
+
bool VPlanTransforms::handleMaxMinNumReductions(VPlan &Plan) {
auto GetMinOrMaxCompareValue =
[](VPReductionPHIRecipe *RedPhiR) -> VPValue * {
@@ -1926,11 +1956,13 @@ bool VPlanTransforms::handleFindLastReductions(VPlan &Plan) {
/// \p FindIVSelect, \p FindIVCmp, and \p FindIVRdxResult, which are replaced
/// and removed.
/// Returns true if the pattern was handled successfully, false otherwise.
-static bool handleFirstArgMinOrMax(
- VPlan &Plan, VPReductionPHIRecipe *MinOrMaxPhiR,
- VPReductionPHIRecipe *FindLastIVPhiR, VPWidenIntOrFpInductionRecipe *WideIV,
- VPInstruction *MinOrMaxResult, VPInstruction *FindIVSelect,
- VPRecipeBase *FindIVCmp, VPInstruction *FindIVRdxResult) {
+static bool
+handleFirstArgMinOrMax(VPlan &Plan, VPReductionPHIRecipe *MinOrMaxPhiR,
+ VPReductionPHIRecipe *FindLastIVPhiR,
+ VPWidenIntOrFpInductionRecipe *WideIV,
+ VPValue *IndexOffset, VPInstruction *MinOrMaxResult,
+ VPInstruction *FindIVSelect, VPRecipeBase *FindIVCmp,
+ VPInstruction *FindIVRdxResult) {
assert(!FindLastIVPhiR->isInLoop() && !FindLastIVPhiR->isOrdered() &&
"inloop and ordered reductions not supported");
assert(FindLastIVPhiR->getVFScaleFactor() == 1 &&
@@ -1947,9 +1979,24 @@ static bool handleFirstArgMinOrMax(
assert(
match(FindIVSelectR, m_Select(m_VPValue(), m_VPValue(), m_VPValue())) &&
"backedge value must be a select");
- if (FindIVSelectR->getOperand(1) != WideIV &&
- FindIVSelectR->getOperand(2) != WideIV)
- return false;
+ // Identify the select arm that stores the index; the other arm is the
+ // reduction phi. Without an offset the stored arm is WideIV itself (the
+ // exact-match requirement bails cleanly on any other expression); with an
+ // offset it is add(WideIV, offset), so match against the phi instead.
+ unsigned StoredIdx;
+ if (IndexOffset) {
+ if (FindIVSelectR->getOperand(1) == FindLastIVPhiR)
+ StoredIdx = 2;
+ else if (FindIVSelectR->getOperand(2) == FindLastIVPhiR)
+ StoredIdx = 1;
+ else
+ return false;
+ } else {
+ if (FindIVSelectR->getOperand(1) != WideIV &&
+ FindIVSelectR->getOperand(2) != WideIV)
+ return false;
+ StoredIdx = FindIVSelectR->getOperand(1) == WideIV ? 1 : 2;
+ }
// If the original wide IV is not canonical, create a new one. The canonical
// wide IV is guaranteed to not wrap for all lanes that are active in the
@@ -1965,8 +2012,7 @@ static bool handleFirstArgMinOrMax(
WidenCanIV->insertBefore(WideIV);
// Update the select to use the wide canonical IV.
- FindIVSelectR->setOperand(FindIVSelectR->getOperand(1) == WideIV ? 1 : 2,
- WidenCanIV);
+ FindIVSelectR->setOperand(StoredIdx, WidenCanIV);
}
FindLastIVPhiR->setOperand(0, Plan.getPoison(Ty));
@@ -2020,10 +2066,20 @@ static bool handleFirstArgMinOrMax(
// vp<%final.idx> = select vp<%always.false>, ir<10>,
// vp<%scaled.idx>
+ // The min/max value comparisons below must use fcmp for FP recurrences; the
+ // index/IV comparisons stay integer. NaN-free semantics (required to reach
+ // here) make the ordered equality exact.
+ bool IsFPMinMax = RecurrenceDescriptor::isFPMinMaxRecurrenceKind(
+ MinOrMaxPhiR->getRecurrenceKind());
+ auto CreateValueEqCmp = [&](VPBuilder &B, VPValue *A, VPValue *C) {
+ return IsFPMinMax ? B.createFCmp(CmpInst::FCMP_OEQ, A, C)
+ : B.createICmp(CmpInst::ICMP_EQ, A, C);
+ };
+
VPBuilder Builder(FindIVRdxResult);
VPValue *MinOrMaxExiting = MinOrMaxResult->getOperand(0);
auto *FinalMinOrMaxCmp =
- Builder.createICmp(CmpInst::ICMP_EQ, MinOrMaxExiting, MinOrMaxResult);
+ CreateValueEqCmp(Builder, MinOrMaxExiting, MinOrMaxResult);
VPValue *LastIVExiting = FindIVRdxResult->getOperand(0);
VPValue *MaxIV =
Plan.getConstantInt(APInt::getMaxValue(Ty->getIntegerBitWidth()));
@@ -2045,11 +2101,19 @@ static bool handleFirstArgMinOrMax(
FinalCanIV = DerivedIVRecipe;
}
+ // Apply the additive offset of a stored IV increment (e.g. iv-1) so the
+ // reconstructed index matches the value the scalar loop stored.
+ if (IndexOffset)
+ FinalCanIV = Builder.createNaryOp(
+ Instruction::Add, {FinalCanIV, IndexOffset},
+ VPIRFlags(VPIRFlags::WrapFlagsTy(/*HasNUW=*/false, /*HasNSW=*/false)),
+ FindIVRdxResult->getDebugLoc());
+
// If the final min/max value matches its start value, the condition in the
// loop was always false, i.e. no induction value has been selected. If that's
// the case, set the result of the IV reduction to its start value.
- VPValue *AlwaysFalse = Builder.createICmp(CmpInst::ICMP_EQ, MinOrMaxResult,
- MinOrMaxPhiR->getStartValue());
+ VPValue *AlwaysFalse =
+ CreateValueEqCmp(Builder, MinOrMaxResult, MinOrMaxPhiR->getStartValue());
VPValue *FinalIV = Builder.createSelect(
AlwaysFalse, FindIVSelect->getOperand(2), FinalCanIV);
FindIVSelect->replaceAllUsesWith(FinalIV);
@@ -2080,8 +2144,10 @@ bool VPlanTransforms::handleMultiUseReductions(VPlan &Plan,
// min/max operation, and be used only by the select of the FindLastIV
// reduction cycle.
RecurKind RdxKind = MinOrMaxPhiR->getRecurrenceKind();
+ bool IsFPMinMax = RecurrenceDescriptor::isFPMinMaxRecurrenceKind(RdxKind);
assert(
- RecurrenceDescriptor::isIntMinMaxRecurrenceKind(RdxKind) &&
+ (RecurrenceDescriptor::isIntMinMaxRecurrenceKind(RdxKind) ||
+ IsFPMinMax) &&
"only min/max recurrences support users outside the reduction chain");
auto *MinOrMaxOp =
@@ -2089,26 +2155,37 @@ bool VPlanTransforms::handleMultiUseReductions(VPlan &Plan,
if (!MinOrMaxOp)
return false;
- // Check that MinOrMaxOp is a VPWidenIntrinsicRecipe or VPReplicateRecipe
- // with an intrinsic that matches the reduction kind.
+ // The value reduction's backedge op is either a min/max intrinsic (the
+ // canonical form) or, for a select-based FP min/max, a select whose
+ // condition is the reduction compare and whose arms are the value operands.
Intrinsic::ID ExpectedIntrinsicID = getMinMaxReductionIntrinsicOp(RdxKind);
- if (!match(MinOrMaxOp, m_Intrinsic(ExpectedIntrinsicID)))
+ bool IsSelectOp =
+ match(MinOrMaxOp, m_Select(m_VPValue(), m_VPValue(), m_VPValue()));
+ if (!match(MinOrMaxOp, m_Intrinsic(ExpectedIntrinsicID)) && !IsSelectOp)
return false;
// MinOrMaxOp must have 2 users: 1) MinOrMaxPhiR and 2)
// ComputeReductionResult.
assert(MinOrMaxOp->getNumUsers() == 2 &&
"MinOrMaxOp must have exactly 2 users");
- VPValue *MinOrMaxOpValue = MinOrMaxOp->getOperand(0);
+ // A select carries its condition in operand 0, so its value operands are 1
+ // and 2; an intrinsic's are 0 and 1.
+ unsigned FirstValOp = IsSelectOp ? 1 : 0;
+ VPValue *MinOrMaxOpValue = MinOrMaxOp->getOperand(FirstValOp);
if (MinOrMaxOpValue == MinOrMaxPhiR)
- MinOrMaxOpValue = MinOrMaxOp->getOperand(1);
+ MinOrMaxOpValue = MinOrMaxOp->getOperand(FirstValOp + 1);
VPValue *CmpOpA;
VPValue *CmpOpB;
CmpPredicate Pred;
auto *Cmp = dyn_cast_or_null<VPRecipeWithIRFlags>(findUserOf(
MinOrMaxPhiR, m_Cmp(Pred, m_VPValue(CmpOpA), m_VPValue(CmpOpB))));
- if (!Cmp || Cmp->getNumUsers() != 1 ||
+ // The shared compare normally feeds only the FindIV select. A down-counting
+ // (FindFirst) FP argmin lowered via the AnyOf path adds an extra Or user,
+ // which is tolerated: the FindIV select is located explicitly below and the
+ // AnyOf scaffolding is torn down once the reductions are combined. Integer
+ // argmin keeps the strict single-user requirement.
+ if (!Cmp || (!IsFPMinMax && Cmp->getNumUsers() != 1) ||
(CmpOpA != MinOrMaxOpValue && CmpOpB != MinOrMaxOpValue))
return false;
@@ -2127,12 +2204,25 @@ bool VPlanTransforms::handleMultiUseReductions(VPlan &Plan,
"one user must be MinOrMaxOp");
assert(MinOrMaxResult && "MinOrMaxResult must be a user of MinOrMaxOp");
- // Cmp must be used by the select of a FindLastIV chain.
- VPValue *Sel = dyn_cast<VPSingleDefRecipe>(Cmp->getSingleUser());
- VPValue *IVOp, *FindIV;
- if (!Sel || Sel->getNumUsers() != 2 ||
- !match(Sel,
- m_Select(m_Specific(Cmp), m_VPValue(IVOp), m_VPValue(FindIV))))
+ // Locate the FindIV select among the compare's users. There is exactly one
+ // select; a down-counting FP AnyOf reduction additionally uses the compare
+ // in an Or, which is tolerated (and cleaned up) below.
+ VPSingleDefRecipe *Sel = nullptr;
+ VPValue *IVOp = nullptr, *FindIV = nullptr;
+ for (VPUser *U : Cmp->users()) {
+ auto *R = dyn_cast<VPSingleDefRecipe>(U);
+ // Skip the value reduction's own select (select-based min/max); only the
+ // index select should be captured here.
+ if (R == MinOrMaxOp)
+ continue;
+ if (R && match(R, m_Select(m_Specific(Cmp), m_VPValue(IVOp),
+ m_VPValue(FindIV)))) {
+ if (Sel)
+ return false;
+ Sel = R;
+ }
+ }
+ if (!Sel || Sel->getNumUsers() != 2)
return false;
if (!isa<VPReductionPHIRecipe>(FindIV)) {
@@ -2148,21 +2238,38 @@ bool VPlanTransforms::handleMultiUseReductions(VPlan &Plan,
assert(!FindIVPhiR->isInLoop() && !FindIVPhiR->isOrdered() &&
"cannot handle inloop/ordered reductions yet");
- // Check if FindIVPhiR is a FindLast pattern by checking the MinMaxKind
- // on its ComputeReductionResult. SMax/UMax indicates FindLast.
+ // A scalar-only VPlan (VF=1) lowers the stored index as a scalar recipe
+ // rather than a widened induction. Such a plan needs no cross-lane
+ // combining, so skip this reduction; bailing would drop the scalar VPlan
+ // and trip a planner assertion once vector plans are built.
+ if (auto *R = IVOp->getDefiningRecipe())
+ if (isa<VPReplicateRecipe, VPScalarIVStepsRecipe>(R))
+ continue;
+
+ // Classify the index reduction: SMax/UMax is FindLast (up-counting),
+ // SMin/UMin is FindFirst (down-counting). Support is scoped to integer
+ // FindLast (upstream) and FP FindFirst.
+ // TODO: FP up-counting (FindLast) and integer down-counting (FindFirst).
VPInstruction *FindIVResult =
findUserOf<VPInstruction::ComputeReductionResult>(
FindIVPhiR->getBackedgeValue());
assert(FindIVResult &&
"must be able to retrieve the FindIVResult VPInstruction");
RecurKind FindIVMinMaxKind = FindIVResult->getRecurKind();
- if (FindIVMinMaxKind != RecurKind::SMax &&
- FindIVMinMaxKind != RecurKind::UMax)
+ bool IsFindLast = !IsFPMinMax && (FindIVMinMaxKind == RecurKind::SMax ||
+ FindIVMinMaxKind == RecurKind::UMax);
+ bool IsFindFirst = IsFPMinMax && (FindIVMinMaxKind == RecurKind::SMin ||
+ FindIVMinMaxKind == RecurKind::UMin);
+ if (!IsFindLast && !IsFindFirst)
return false;
- // TODO: Support cases where IVOp is the IV increment.
- if (!match(IVOp, m_TruncOrSelf(m_VPValue(IVOp))) ||
- !isa<VPWidenIntOrFpInductionRecipe>(IVOp))
+ // The stored index is the induction WideIV, or (for a down-counting loop
+ // storing the IV increment, e.g. iv-1) add(WideIV, invariant). Recover the
+ // base induction and any additive offset to reapply when the final index is
+ // reconstructed. Offsets are only accepted for FindFirst.
+ VPValue *IndexOffset = nullptr;
+ auto *WideIV = getFindIVBaseInduction(IVOp, IndexOffset);
+ if (!WideIV || (IndexOffset && !IsFindFirst))
return false;
// Check if the predicate is compatible with the reduction kind.
@@ -2176,8 +2283,18 @@ bool VPlanTransforms::handleMultiUseReductions(VPlan &Plan,
return Pred == CmpInst::ICMP_SLE || Pred == CmpInst::ICMP_SLT;
case RecurKind::SMin:
return Pred == CmpInst::ICMP_SGE || Pred == CmpInst::ICMP_SGT;
+ case RecurKind::FMin:
+ case RecurKind::FMinNum:
+ return Pred == CmpInst::FCMP_OGE || Pred == CmpInst::FCMP_OGT ||
+ Pred == CmpInst::FCMP_UGE || Pred == CmpInst::FCMP_UGT;
+ case RecurKind::FMax:
+ case RecurKind::FMaxNum:
+ return Pred == CmpInst::FCMP_OLE || Pred == CmpInst::FCMP_OLT ||
+ Pred == CmpInst::FCMP_ULE || Pred == CmpInst::FCMP_ULT;
default:
- llvm_unreachable("unhandled recurrence kind");
+ // FMinimum/FMaximum and their *Num variants have different NaN and
+ // signed-zero semantics; do not combine them with an index reduction.
+ return false;
}
}();
if (!IsValidKindPred) {
@@ -2192,27 +2309,86 @@ bool VPlanTransforms::handleMultiUseReductions(VPlan &Plan,
return false;
}
- auto *FindIVSelect = findFindIVSelect(FindIVPhiR->getBackedgeValue());
- auto *FindIVCmp = FindIVSelect->getOperand(0)->getDefiningRecipe();
- auto *FindIVRdxResult = cast<VPInstruction>(FindIVCmp->getOperand(0));
+ VPInstruction *FindIVSelect = nullptr;
+ VPRecipeBase *FindIVCmp = nullptr;
+ VPInstruction *FindIVRdxResult = nullptr;
+ if (IsFindFirst) {
+ // For down-counting loops sinking is disabled for the multi-use argmin,
+ // so the min/max index reduction result feeds the middle-block select
+ // directly, for either lowering:
+ // sentinel: select(icmp ne <rdx>, Sentinel), <rdx>, Start
+ // AnyOf: select(freeze(<or-reduce>), <rdx>, Start
+ FindIVRdxResult = FindIVResult;
+ for (VPUser *U : FindIVRdxResult->users()) {
+ auto *R = dyn_cast<VPInstruction>(U);
+ if (R && R->getOpcode() == Instruction::Select &&
+ R->getOperand(1) == FindIVRdxResult) {
+ FindIVSelect = R;
+ break;
+ }
+ }
+ if (!FindIVSelect)
+ return false;
+ FindIVCmp = FindIVSelect->getOperand(0)->getDefiningRecipe();
+ if (!FindIVCmp)
+ return false;
+ } else {
+ FindIVSelect = findFindIVSelect(FindIVPhiR->getBackedgeValue());
+ FindIVCmp = FindIVSelect->getOperand(0)->getDefiningRecipe();
+ FindIVRdxResult = cast<VPInstruction>(FindIVCmp->getOperand(0));
+ }
assert(FindIVSelect->getParent() == MinOrMaxResult->getParent() &&
"both results must be computed in the same block");
+
+ // For the AnyOf lowering FindIVCmp is a freeze of an Or reduction. Capture
+ // that Or reduction result now so its scaffolding can be erased after the
+ // final index is rebuilt below.
+ auto *FindIVCmpI = dyn_cast<VPInstruction>(FindIVCmp);
+ bool IsAnyOf = FindIVCmpI && FindIVCmpI->getOpcode() == Instruction::Freeze;
+ VPValue *OrReduceVal = IsAnyOf ? FindIVCmpI->getOperand(0) : nullptr;
// Reducing to a scalar min or max value is placed right before reducing to
// its scalar iteration, in order to generate instructions that use both
// their operands.
MinOrMaxResult->moveBefore(*FindIVRdxResult->getParent(),
FindIVRdxResult->getIterator());
- bool IsStrictPredicate = ICmpInst::isLT(Pred) || ICmpInst::isGT(Pred);
+ bool IsStrictPredicate =
+ CmpInst::isFPPredicate(Pred)
+ ? (Pred == CmpInst::FCMP_OLT || Pred == CmpInst::FCMP_OGT ||
+ Pred == CmpInst::FCMP_ULT || Pred == CmpInst::FCMP_UGT)
+ : (ICmpInst::...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/209827
More information about the llvm-branch-commits
mailing list