[llvm] [VPlan] Fix VPInstruction::AnyOf combine undoing freeze (PR #223640)
via llvm-commits
llvm-commits at lists.llvm.org
Tue Sep 15 02:46:11 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-vectorizers
Author: Luke Lau (lukel97)
<details>
<summary>Changes</summary>
There is an any-of combine for unrolled VPlans which does:
any-of (fcmp uno A, A), (fcmp uno B, B), ...-> any-of (fcmp uno A, B)
However any-of implicitly freezes each individual operand and this means we go from `freeze (fcmp uno A, A)` to `freeze (fcmp uno A, B)` which isn't sound: https://alive2.llvm.org/ce/z/UdQM7C
This causes miscompiles today with early exit loops, see the attached test case in single-early-exit-anyof-fold.ll.
This fixes it by explicitly modelling the freeze in VPlan. There are three places where we use AnyOf:
1) early exit loops: the freeze needs to be applied per-lane, so apply it to `(any-of (freeze (combined-conds-to-exit)))`
2) handleMaxMinNumReductions: If any lane of the reduction was poison in the scalar loop, the final result will be poison. We only need to freeze the result of AnyOf to prevent immediate UB when branching. Freezing individual operands blocks the any-of combine otherwise.
3) handleFindLastReductions: we need to freeze Cond itself since it's got multiple uses, but I plan on fixing this in a separate PR
---
Patch is 131.83 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/223640.diff
27 Files Affected:
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h (+5)
- (modified) llvm/lib/Transforms/Vectorize/VPlan.h (+1-2)
- (modified) llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp (+6-2)
- (modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+9-3)
- (modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp (+13-5)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/cmp_cost.ll (+51-48)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/fmax-without-fast-math-flags.ll (+28-32)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/fmin-without-fast-math-flags.ll (+11-12)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/single-early-exit-interleave.ll (+2-2)
- (modified) llvm/test/Transforms/LoopVectorize/AArch64/sve-non-native-interleaved-accesses.ll (+2-3)
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/conditional-scalar-assignment-vplan.ll (+35-31)
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/early_exit_with_stores_vplan.ll (+30-27)
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/interleave-conditional-scalar-assignment-vplan.ll (+31-27)
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/uncountable-early-exit-vplan.ll (+19-15)
- (modified) llvm/test/Transforms/LoopVectorize/X86/cost-any-of.ll (+10-11)
- (modified) llvm/test/Transforms/LoopVectorize/epilog-vectorization-fmaxnum-reductions.ll (+6-6)
- (modified) llvm/test/Transforms/LoopVectorize/fcmp-uno-fold-interleave.ll (+32-36)
- (modified) llvm/test/Transforms/LoopVectorize/fmax-without-fast-math-flags-interleave.ll (+15-16)
- (modified) llvm/test/Transforms/LoopVectorize/fmax-without-fast-math-flags.ll (+12-12)
- (modified) llvm/test/Transforms/LoopVectorize/fmin-without-fast-math-flags.ll (+4-4)
- (modified) llvm/test/Transforms/LoopVectorize/iv-select-cmp-non-const-iv-start.ll (+4-4)
- (modified) llvm/test/Transforms/LoopVectorize/iv-select-cmp-trunc.ll (+24-24)
- (modified) llvm/test/Transforms/LoopVectorize/iv-select-cmp.ll (+2-2)
- (modified) llvm/test/Transforms/LoopVectorize/select-cmp.ll (+8-8)
- (added) llvm/test/Transforms/LoopVectorize/single-early-exit-anyof-fold.ll (+57)
- (modified) llvm/test/Transforms/LoopVectorize/single-early-exit-interleave-hint.ll (+2-2)
- (modified) llvm/test/Transforms/LoopVectorize/single-early-exit-interleave.ll (+18-18)
``````````diff
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index 9b6f0f37008a68..b89eead99b9470 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -289,6 +289,11 @@ class VPBuilder {
Name);
}
+ VPInstruction *createFreeze(VPValue *Op, DebugLoc DL = DebugLoc::getUnknown(),
+ const Twine &Name = "") {
+ return createNaryOp(Instruction::Freeze, Op, DL, Name);
+ }
+
VPInstruction *createLogicalAnd(VPValue *LHS, VPValue *RHS,
DebugLoc DL = DebugLoc::getUnknown(),
const Twine &Name = "") {
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index daabca902724b1..d570002fb82367 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -1378,8 +1378,7 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
// Returns a scalar boolean value, which is true if any lane of its
// (boolean) vector operands is true. It produces the reduced value across
// all unrolled iterations. Unrolling will add all copies of its original
- // operand as additional operands. AnyOf is poison-safe as all operands
- // will be frozen.
+ // operand as additional operands.
AnyOf,
// Calculates the first active lane index of the vector predicate operands.
// It produces the lane index across all unrolled iterations. Unrolling will
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 4cd455adb7a902..4b1ebed273dcff 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -1695,7 +1695,9 @@ bool VPlanTransforms::handleMaxMinNumReductions(VPlan &Plan) {
}
VPValue *AnyNaNLane =
- LatchBuilder.createNaryOp(VPInstruction::AnyOf, {AllNaNLanes});
+ LatchBuilder.createNaryOp(VPInstruction::AnyOf, AllNaNLanes);
+ // Freeze to prevent immediate UB from branching on poison.
+ AnyNaNLane = LatchBuilder.createFreeze(AnyNaNLane);
VPBasicBlock *MiddleVPBB = Plan.getMiddleBlock();
VPBuilder MiddleBuilder(MiddleVPBB, MiddleVPBB->begin());
for (const auto &[RedPhiR, _] : MinOrMaxNumReductionsToHandle) {
@@ -1874,7 +1876,9 @@ bool VPlanTransforms::handleFindLastReductions(VPlan &Plan) {
if (HeaderMask)
Cond = Builder.createLogicalAnd(HeaderMask, Cond);
- VPValue *AnyOf = Builder.createNaryOp(VPInstruction::AnyOf, {Cond});
+ VPValue *AnyOf =
+ Builder.createNaryOp(VPInstruction::AnyOf, Builder.createFreeze(Cond));
+ // FIXME: The Cond here needs to be frozen too.
VPValue *MaskSelect = Builder.createSelect(AnyOf, Cond, MaskPHI);
MaskPHI->addIncoming(MaskSelect);
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 5997b6e0fef76d..4f3699e7cb8028 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -1036,9 +1036,9 @@ Value *VPInstruction::generate(VPTransformState &State) {
return Builder.CreatePtrAdd(Ptr, Addend, Name, getGEPNoWrapFlags());
}
case VPInstruction::AnyOf: {
- Value *Res = Builder.CreateFreeze(State.get(getOperand(0)));
+ Value *Res = State.get(getOperand(0));
for (VPValue *Op : drop_begin(operands()))
- Res = Builder.CreateOr(Res, Builder.CreateFreeze(State.get(Op)));
+ Res = Builder.CreateOr(Res, State.get(Op));
return State.VF.isScalar() ? Res : Builder.CreateOrReduce(Res);
}
case VPInstruction::ExtractLane: {
@@ -1741,9 +1741,15 @@ bool VPInstruction::usesFirstLaneOnly(const VPValue *Op) const {
return Op == getOperand(1) || Op == getOperand(2);
case Instruction::PHI:
return true;
+ case Instruction::Select:
+ if (Op == getOperand(0) && Op != getOperand(1) && Op != getOperand(2) &&
+ vputils::isSingleScalar(Op))
+ // TODO: Relax for other opcodes.
+ if (match(Op, m_Freeze(m_VPValue())))
+ return true;
+ [[fallthrough]];
case Instruction::FCmp:
case Instruction::ICmp:
- case Instruction::Select:
case Instruction::Or:
case Instruction::Freeze:
case VPInstruction::Not:
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index 5b95f5948cf7de..5b98875f125fd2 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -3064,9 +3064,10 @@ getRecipesForUncountableExit(SmallVectorImpl<VPInstruction *> &Recipes,
// Find the uncountable loop exit condition.
VPValue *UncountableCondition = nullptr;
- if (!match(LatchVPBB->getTerminator(),
- m_BranchOnTwoConds(m_AnyOf(m_VPValue(UncountableCondition)),
- m_VPValue())))
+ if (!match(
+ LatchVPBB->getTerminator(),
+ m_BranchOnTwoConds(m_AnyOf(m_Freeze(m_VPValue(UncountableCondition))),
+ m_VPValue())))
return nullptr;
SmallVector<VPValue *, 4> Worklist;
@@ -3385,8 +3386,15 @@ bool VPlanTransforms::handleUncountableEarlyExits(
for (const EarlyExitInfo &Info : drop_begin(Exits))
Combined = LatchBuilder.createLogicalOr(Combined, Info.CondToExit);
- VPValue *IsAnyExitTaken =
- LatchBuilder.createNaryOp(VPInstruction::AnyOf, {Combined});
+ // Even though the logical or prevents per-lane posion propagation, we need to
+ // freeze Combined to prevent poisoning the entire AnyOf result:
+ //
+ // Exits[0].CondToExit = [0,1,0,0]
+ // Exits[1].CondToExit = [0,0,p,p]
+ // Combined = [0,1,p,p]
+ // AnyOf = 1
+ VPValue *IsAnyExitTaken = LatchBuilder.createNaryOp(
+ VPInstruction::AnyOf, LatchBuilder.createFreeze(Combined));
// Create a comparison for the latch exit condition and replace the
// BranchOnCond with a BranchOnTwoConds. The original BranchOnCond's condition
diff --git a/llvm/test/Transforms/LoopVectorize/AArch64/cmp_cost.ll b/llvm/test/Transforms/LoopVectorize/AArch64/cmp_cost.ll
index a25edccb605145..fee76962809427 100644
--- a/llvm/test/Transforms/LoopVectorize/AArch64/cmp_cost.ll
+++ b/llvm/test/Transforms/LoopVectorize/AArch64/cmp_cost.ll
@@ -19,13 +19,14 @@ define float @fmaxnum_reduction_f32(float %base, i32 %n) {
; CHECK: Cost of 0 for VF 2: EMIT vp<%index.next> = add nuw vp<[[VP3:%[0-9]+]]>, vp<[[VP1:%[0-9]+]]>
; CHECK: Cost of 3 for VF 2: EMIT vp<[[VP4:%[0-9]+]]> = fcmp uno ir<%v>, ir<%v>
; CHECK: Cost of 2 for VF 2: EMIT vp<[[VP5:%[0-9]+]]> = any-of vp<[[VP4]]>
-; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP6:%[0-9]+]]> = icmp eq vp<%index.next>, vp<[[VP2:%[0-9]+]]>
-; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP7:%[0-9]+]]> = or vp<[[VP5]]>, vp<[[VP6]]>
-; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP7]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP6:%[0-9]+]]> = freeze vp<[[VP5]]>
+; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP7:%[0-9]+]]> = icmp eq vp<%index.next>, vp<[[VP2:%[0-9]+]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP8:%[0-9]+]]> = or vp<[[VP6]]>, vp<[[VP7]]>
+; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP8]]>
; CHECK: Cost of 0 for VF 2: vector loop backedge
; CHECK: Cost of 1 for VF 2: canonical IV increment
-; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.resume.val> = phi [ vp<[[VP10:%[0-9]+]]>, middle.block ], [ ir<0>, ir-bb<entry> ]
-; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.merge.rdx> = phi [ vp<[[VP11:%[0-9]+]]>, middle.block ], [ ir<-1.000000e+07>, ir-bb<entry> ]
+; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.resume.val> = phi [ vp<[[VP11:%[0-9]+]]>, middle.block ], [ ir<0>, ir-bb<entry> ]
+; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.merge.rdx> = phi [ vp<[[VP12:%[0-9]+]]>, middle.block ], [ ir<-1.000000e+07>, ir-bb<entry> ]
; CHECK: Cost of 0 for VF 2: IR %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ] (extra operand: vp<%bc.resume.val> from scalar.ph)
; CHECK: Cost of 0 for VF 2: IR %max = phi float [ -1.000000e+07, %entry ], [ %max.next, %loop ] (extra operand: vp<%bc.merge.rdx> from scalar.ph)
; CHECK: Cost of 0 for VF 2: IR %iv.f = sitofp i32 %iv to float
@@ -33,14 +34,14 @@ define float @fmaxnum_reduction_f32(float %base, i32 %n) {
; CHECK: Cost of 0 for VF 2: IR %max.next = call float @llvm.maxnum.f32(float %max, float %v)
; CHECK: Cost of 0 for VF 2: IR %iv.next = add nuw nsw i32 %iv, 1
; CHECK: Cost of 0 for VF 2: IR %ec = icmp eq i32 %iv.next, %n
-; CHECK: Cost of 2 for VF 2: EMIT vp<[[VP9:%[0-9]+]]> = select vp<[[VP5]]>, ir<%max>, ir<%max.next>
-; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP10]]> = select vp<[[VP5]]>, vp<[[VP3]]>, vp<[[VP2]]>
-; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP11]]> = compute-reduction-result (fmaxnum) vp<[[VP9]]>
+; CHECK: Cost of 2 for VF 2: EMIT vp<[[VP10:%[0-9]+]]> = select vp<[[VP6]]>, ir<%max>, ir<%max.next>
+; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP11]]> = select vp<[[VP6]]>, vp<[[VP3]]>, vp<[[VP2]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP12]]> = compute-reduction-result (fmaxnum) vp<[[VP10]]>
; CHECK: Cost of 1 for VF 2: EMIT vp<%cmp.n> = icmp eq ir<%n>, vp<[[VP2]]>
-; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP12:%[0-9]+]]> = not vp<[[VP5]]>
-; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP13:%[0-9]+]]> = and vp<%cmp.n>, vp<[[VP12]]>
-; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP13]]>
-; CHECK: Cost of 0 for VF 2: IR %max.next.lcssa = phi float [ %max.next, %loop ] (extra operand: vp<[[VP11]]> from middle.block)
+; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP13:%[0-9]+]]> = not vp<[[VP6]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP14:%[0-9]+]]> = and vp<%cmp.n>, vp<[[VP13]]>
+; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP14]]>
+; CHECK: Cost of 0 for VF 2: IR %max.next.lcssa = phi float [ %max.next, %loop ] (extra operand: vp<[[VP12]]> from middle.block)
; CHECK: Cost of 1 for VF 4: ir<%iv> = WIDEN-INDUCTION nuw nsw ir<0>, ir<1>, vp<[[VP0]]>
; CHECK: Cost of 0 for VF 4: WIDEN-REDUCTION-PHI ir<%max> = phi (fmaxnum) ir<-1.000000e+07>, ir<%max.next>
; CHECK: Cost of 1 for VF 4: WIDEN-CAST ir<%iv.f> = sitofp ir<%iv> to float
@@ -49,13 +50,14 @@ define float @fmaxnum_reduction_f32(float %base, i32 %n) {
; CHECK: Cost of 0 for VF 4: EMIT vp<%index.next> = add nuw vp<[[VP3]]>, vp<[[VP1]]>
; CHECK: Cost of 3 for VF 4: EMIT vp<[[VP4]]> = fcmp uno ir<%v>, ir<%v>
; CHECK: Cost of 2 for VF 4: EMIT vp<[[VP5]]> = any-of vp<[[VP4]]>
-; CHECK: Cost of 1 for VF 4: EMIT vp<[[VP6]]> = icmp eq vp<%index.next>, vp<[[VP2]]>
-; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP7]]> = or vp<[[VP5]]>, vp<[[VP6]]>
-; CHECK: Cost of 0 for VF 4: EMIT branch-on-cond vp<[[VP7]]>
+; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP6]]> = freeze vp<[[VP5]]>
+; CHECK: Cost of 1 for VF 4: EMIT vp<[[VP7]]> = icmp eq vp<%index.next>, vp<[[VP2]]>
+; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP8]]> = or vp<[[VP6]]>, vp<[[VP7]]>
+; CHECK: Cost of 0 for VF 4: EMIT branch-on-cond vp<[[VP8]]>
; CHECK: Cost of 0 for VF 4: vector loop backedge
; CHECK: Cost of 1 for VF 4: canonical IV increment
-; CHECK: Cost of 0 for VF 4: EMIT-SCALAR vp<%bc.resume.val> = phi [ vp<[[VP10]]>, middle.block ], [ ir<0>, ir-bb<entry> ]
-; CHECK: Cost of 0 for VF 4: EMIT-SCALAR vp<%bc.merge.rdx> = phi [ vp<[[VP11]]>, middle.block ], [ ir<-1.000000e+07>, ir-bb<entry> ]
+; CHECK: Cost of 0 for VF 4: EMIT-SCALAR vp<%bc.resume.val> = phi [ vp<[[VP11]]>, middle.block ], [ ir<0>, ir-bb<entry> ]
+; CHECK: Cost of 0 for VF 4: EMIT-SCALAR vp<%bc.merge.rdx> = phi [ vp<[[VP12]]>, middle.block ], [ ir<-1.000000e+07>, ir-bb<entry> ]
; CHECK: Cost of 0 for VF 4: IR %iv = phi i32 [ 0, %entry ], [ %iv.next, %loop ] (extra operand: vp<%bc.resume.val> from scalar.ph)
; CHECK: Cost of 0 for VF 4: IR %max = phi float [ -1.000000e+07, %entry ], [ %max.next, %loop ] (extra operand: vp<%bc.merge.rdx> from scalar.ph)
; CHECK: Cost of 0 for VF 4: IR %iv.f = sitofp i32 %iv to float
@@ -63,14 +65,14 @@ define float @fmaxnum_reduction_f32(float %base, i32 %n) {
; CHECK: Cost of 0 for VF 4: IR %max.next = call float @llvm.maxnum.f32(float %max, float %v)
; CHECK: Cost of 0 for VF 4: IR %iv.next = add nuw nsw i32 %iv, 1
; CHECK: Cost of 0 for VF 4: IR %ec = icmp eq i32 %iv.next, %n
-; CHECK: Cost of 2 for VF 4: EMIT vp<[[VP9]]> = select vp<[[VP5]]>, ir<%max>, ir<%max.next>
-; CHECK: Cost of 1 for VF 4: EMIT vp<[[VP10]]> = select vp<[[VP5]]>, vp<[[VP3]]>, vp<[[VP2]]>
-; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP11]]> = compute-reduction-result (fmaxnum) vp<[[VP9]]>
+; CHECK: Cost of 2 for VF 4: EMIT vp<[[VP10]]> = select vp<[[VP6]]>, ir<%max>, ir<%max.next>
+; CHECK: Cost of 1 for VF 4: EMIT vp<[[VP11]]> = select vp<[[VP6]]>, vp<[[VP3]]>, vp<[[VP2]]>
+; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP12]]> = compute-reduction-result (fmaxnum) vp<[[VP10]]>
; CHECK: Cost of 1 for VF 4: EMIT vp<%cmp.n> = icmp eq ir<%n>, vp<[[VP2]]>
-; CHECK: Cost of 1 for VF 4: EMIT vp<[[VP12]]> = not vp<[[VP5]]>
-; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP13]]> = and vp<%cmp.n>, vp<[[VP12]]>
-; CHECK: Cost of 0 for VF 4: EMIT branch-on-cond vp<[[VP13]]>
-; CHECK: Cost of 0 for VF 4: IR %max.next.lcssa = phi float [ %max.next, %loop ] (extra operand: vp<[[VP11]]> from middle.block)
+; CHECK: Cost of 1 for VF 4: EMIT vp<[[VP13]]> = not vp<[[VP6]]>
+; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP14]]> = and vp<%cmp.n>, vp<[[VP13]]>
+; CHECK: Cost of 0 for VF 4: EMIT branch-on-cond vp<[[VP14]]>
+; CHECK: Cost of 0 for VF 4: IR %max.next.lcssa = phi float [ %max.next, %loop ] (extra operand: vp<[[VP12]]> from middle.block)
; CHECK: LV: Selecting VF: 4.
;
entry:
@@ -100,13 +102,14 @@ define double @fmaxnum_reduction_f64(double %base, i64 %n) {
; CHECK: Cost of 0 for VF 2: EMIT vp<%index.next> = add nuw vp<[[VP3:%[0-9]+]]>, vp<[[VP1:%[0-9]+]]>
; CHECK: Cost of 3 for VF 2: EMIT vp<[[VP4:%[0-9]+]]> = fcmp uno ir<%v>, ir<%v>
; CHECK: Cost of 2 for VF 2: EMIT vp<[[VP5:%[0-9]+]]> = any-of vp<[[VP4]]>
-; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP6:%[0-9]+]]> = icmp eq vp<%index.next>, vp<[[VP2:%[0-9]+]]>
-; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP7:%[0-9]+]]> = or vp<[[VP5]]>, vp<[[VP6]]>
-; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP7]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP6:%[0-9]+]]> = freeze vp<[[VP5]]>
+; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP7:%[0-9]+]]> = icmp eq vp<%index.next>, vp<[[VP2:%[0-9]+]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP8:%[0-9]+]]> = or vp<[[VP6]]>, vp<[[VP7]]>
+; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP8]]>
; CHECK: Cost of 0 for VF 2: vector loop backedge
; CHECK: Cost of 1 for VF 2: canonical IV increment
-; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.resume.val> = phi [ vp<[[VP10:%[0-9]+]]>, middle.block ], [ ir<0>, ir-bb<entry> ]
-; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.merge.rdx> = phi [ vp<[[VP11:%[0-9]+]]>, middle.block ], [ ir<-1.000000e+07>, ir-bb<entry> ]
+; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.resume.val> = phi [ vp<[[VP11:%[0-9]+]]>, middle.block ], [ ir<0>, ir-bb<entry> ]
+; CHECK: Cost of 0 for VF 2: EMIT-SCALAR vp<%bc.merge.rdx> = phi [ vp<[[VP12:%[0-9]+]]>, middle.block ], [ ir<-1.000000e+07>, ir-bb<entry> ]
; CHECK: Cost of 0 for VF 2: IR %iv = phi i64 [ 0, %entry ], [ %iv.next, %loop ] (extra operand: vp<%bc.resume.val> from scalar.ph)
; CHECK: Cost of 0 for VF 2: IR %max = phi double [ -1.000000e+07, %entry ], [ %max.next, %loop ] (extra operand: vp<%bc.merge.rdx> from scalar.ph)
; CHECK: Cost of 0 for VF 2: IR %iv.f = sitofp i64 %iv to double
@@ -114,14 +117,14 @@ define double @fmaxnum_reduction_f64(double %base, i64 %n) {
; CHECK: Cost of 0 for VF 2: IR %max.next = call double @llvm.maxnum.f64(double %max, double %v)
; CHECK: Cost of 0 for VF 2: IR %iv.next = add nuw nsw i64 %iv, 1
; CHECK: Cost of 0 for VF 2: IR %ec = icmp eq i64 %iv.next, %n
-; CHECK: Cost of 2 for VF 2: EMIT vp<[[VP9:%[0-9]+]]> = select vp<[[VP5]]>, ir<%max>, ir<%max.next>
-; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP10]]> = select vp<[[VP5]]>, vp<[[VP3]]>, vp<[[VP2]]>
-; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP11]]> = compute-reduction-result (fmaxnum) vp<[[VP9]]>
+; CHECK: Cost of 2 for VF 2: EMIT vp<[[VP10:%[0-9]+]]> = select vp<[[VP6]]>, ir<%max>, ir<%max.next>
+; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP11]]> = select vp<[[VP6]]>, vp<[[VP3]]>, vp<[[VP2]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP12]]> = compute-reduction-result (fmaxnum) vp<[[VP10]]>
; CHECK: Cost of 1 for VF 2: EMIT vp<%cmp.n> = icmp eq ir<%n>, vp<[[VP2]]>
-; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP12:%[0-9]+]]> = not vp<[[VP5]]>
-; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP13:%[0-9]+]]> = and vp<%cmp.n>, vp<[[VP12]]>
-; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP13]]>
-; CHECK: Cost of 0 for VF 2: IR %max.next.lcssa = phi double [ %max.next, %loop ] (extra operand: vp<[[VP11]]> from middle.block)
+; CHECK: Cost of 1 for VF 2: EMIT vp<[[VP13:%[0-9]+]]> = not vp<[[VP6]]>
+; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP14:%[0-9]+]]> = and vp<%cmp.n>, vp<[[VP13]]>
+; CHECK: Cost of 0 for VF 2: EMIT branch-on-cond vp<[[VP14]]>
+; CHECK: Cost of 0 for VF 2: IR %max.next.lcssa = phi double [ %max.next, %loop ] (extra operand: vp<[[VP12]]> from middle.block)
; CHECK: LV: Selecting VF: 1.
;
entry:
@@ -182,8 +185,8 @@ define i32 @switch_to_cmp(ptr %s, ptr %dst, i64 %n) {
; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP34:%[0-9]+]]> = or vp<[[VP33]]>, vp<[[VP19]]>
; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP35:%[0-9]+]]> = or vp<[[VP34]]>, vp<[[VP20]]>
; CHECK: Cost of 0 for VF 2: EMIT vp<[[VP36:%[0-9]+]]> = or vp<[[VP35]]>, vp<[[VP21]]>
-; CHECK: Cost of 1 for VF 2: WIDEN ir<%c.4> = add ir<%c>, ir<4>
-; CHECK: Cost of 1 for VF 2: WIDEN ir<%c.1> = add ir<%c>, ir<1>
+; CHECK: Cost of 1 for VF 2: WIDEN ir<%c.4> = add ir<%c>, ir<4> (!vplan.execution.frequency 542551298306736128 (5.882%, estimated))
+; CHECK: Cost of 1 for VF 2: WIDEN ir<%c.1> = add ir<%c>, ir<1> (!vplan.execution.frequency 8138269444536270848 (88.24%, estimated))
; CHECK: Cost of 7 for VF 2: BLEND ir<%c.next> = ir<%c> ir<%c.4>/vp<[[VP22]]> ir<%c.1>/vp<[[VP36]]>
; CHECK: Cost of 0 for VF 2: CLONE ir<%dst.gep> = getelementptr ir<%dst>, vp<[[VP5]]>
; CHECK: Cost of 0 for VF 2: vp<[[VP37:%[0-9]+]]> = vector-pointer i8, ir<%dst.gep>, ir<1>
@@ -240,8 +243,8 @@ define i32 @switch_to_cmp(ptr %s, ptr %dst, i64 %n) {
; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP34]]> = or vp<[[VP33]]>, vp<[[VP19]]>
; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP35]]> = or vp<[[VP34]]>, vp<[[VP20]]>
; CHECK: Cost of 0 for VF 4: EMIT vp<[[VP36]]> = or vp<[[VP35]]>, vp<[[VP21]]>
-; CHECK: Cost of 1 for VF 4: WIDEN ir<%c.4> = add ir<%c>, ir<4>
-; CHECK: Cost of 1 for VF 4: WIDEN ir<%c.1> = add ir<%c>, ir<1>
+; CHECK: Cost of 1 for VF 4: WIDEN ir<%c.4> = add ir<%c>, ir<4> (!vplan.execution.frequency 542551298306736128 (5.882%, estimated))
+; CHECK: Cost of 1 for VF 4: WIDEN ir<%c.1> = add ir<%c>, ir<1> (!vplan.execution.frequency 8138269444536270848 (88.24%, estimated))
; CHECK: Cost of 13 for VF 4: BLEND ir<%c.next> = ir<%c> ir<%c.4>/vp<[[VP22]]> ir<%c.1>/vp<[[VP36]]>
; CHECK: Cost of 0 for VF 4: CLONE ir<%dst.gep> = getelementptr ir<%dst>, vp<[[VP5]]>
; CHECK: Cost of 0 for VF 4: vp<[[VP37]]> = vector-pointer i8, ir<%dst.gep>, ir<1>
@@ -298,8 +301,8 @@ define i32 @switch_to_cmp(ptr %s, ptr %dst, i64 %n) {
; CHECK: Cost of 0 for VF 8: EMIT vp<[[VP34]]> = or vp<[[VP33]]>, vp<[[VP19]]>
; CHECK: Cost of 0 for VF 8: EMIT vp<[[VP35]]> = or vp<[[VP34]]>, vp<[[VP20]]>
; CHECK: Cost of 0 for VF 8: EMIT vp<[[VP36]]> = or vp<[[VP35]]>, vp<[[VP21]]>
-; CHECK: Cost of 2 for VF 8: WIDEN ir<%c.4> = add ir<%c>, ir<4>
-; CHECK: Cost of 2 for VF 8: WIDEN ir<%c.1> = add ir<%c>, ir<1>
+; CHECK: Cost of 2 for VF 8: WIDEN ir<%c.4> = add ir<%c>, ir<4> (!vplan.execution.frequency 542551298306736128 (5.882%, estimated))
+; CHECK: Cost of 2 for VF 8: WIDEN ir<%c.1> = add ir<%c>, ir<1> (!vplan.execution.frequency 8138269444536270848 (88.24%, estimated))
; CHECK: Cost of 10 for VF 8: BLEND ir<%c.next> = ir<%c> ir<%c.4>/vp<[[VP22]]> ir<%c.1>/vp<[[VP36]]>
; CHECK: Cost of 0 for VF 8: CLONE ir<%dst.gep> = getelemen...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/223640
More information about the llvm-commits
mailing list