[llvm] [LV] Vectorize predicated loop-invariant loads via masked load + broadcast (PR #200361)
Ashutosh Nema via llvm-commits
llvm-commits at lists.llvm.org
Wed Jul 29 06:06:22 PDT 2026
https://github.com/nema-ashutosh updated https://github.com/llvm/llvm-project/pull/200361
>From 353bb51d29dc345328d8a53ad0b16475c8856f37 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Fri, 29 May 2026 15:04:07 +0530
Subject: [PATCH 1/5] [LV] Vectorize predicated loop-invariant loads via masked
load + broadcast
Today, a predicated load from a loop-invariant address is scalarised
inside a replicate region: every active lane independently issues the
same load and the results are reassembled into a vector. The cost
model assigns this a very high cost, which both blocks
auto-vectorization
and produces poor codegen when vectorization is forced.
This patch teaches the loop vectorizer to instead emit a single
masked load of one lane from the invariant pointer, guarded by an
any-active-lane reduction over the block mask, and broadcast the
result across the vector.
* New cost-model decision CM_CondInvar and helper
getConditionalInvarCost, picked when it beats gather/scatter
and scalarisation.
* New VPInstruction opcode BroadcastLane (dual of ExtractLane),
with type inference, cost, printing and a verifier check.
* New VPlan transform cseUniformMemoryReads, to fold the
structurally identical masked loads produced for each UF part
(VPWidenLoadRecipe is missed by the generic cse).
* Gated by -enable-masked-invariant-load (off by default).
Fixes #200355.
---
.../Transforms/Vectorize/LoopVectorize.cpp | 105 ++++++++++-
.../Transforms/Vectorize/VPRecipeBuilder.h | 5 +
llvm/lib/Transforms/Vectorize/VPlan.h | 3 +
.../lib/Transforms/Vectorize/VPlanRecipes.cpp | 56 ++++++
.../Transforms/Vectorize/VPlanTransforms.cpp | 36 ++++
.../Transforms/Vectorize/VPlanTransforms.h | 4 +
.../Transforms/Vectorize/VPlanVerifier.cpp | 30 ++++
.../enable-masked-invariant-load-multiple.ll | 93 ++++++++++
...ble-masked-invariant-load-not-invariant.ll | 64 +++++++
.../X86/enable-masked-invariant-load.ll | 169 ++++++++++++++++++
10 files changed, 563 insertions(+), 2 deletions(-)
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index aa099b84ed61b..0084bf0ce3a03 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -206,6 +206,10 @@ static cl::opt<bool> ForcePartialAliasingVectorization(
"force-partial-aliasing-vectorization", cl::init(false), cl::Hidden,
cl::desc("Replace pointer diff checks with alias masks."));
+static cl::opt<bool> EnableMaskedInvariantLoad(
+ "enable-masked-invariant-load", cl::init(false), cl::ReallyHidden,
+ cl::desc("Vectorize conditional invariant load with mask-load"
+ " and broadcast"));
/// Option tail-folding-policy controls the tail-folding strategy and lists all
/// available options. The vectorizer will attempt to fold the tail-loop into
/// the vector loop (main/epilogue loops) and predicate the instructions
@@ -882,7 +886,8 @@ class LoopVectorizationCostModel {
/// A widening decision that has been invalidated after replacing the
/// corresponding recipe during VPlan transforms.
/// TODO: Remove once the legacy exit cost computation is retired.
- CM_InvalidatedDecision
+ CM_InvalidatedDecision,
+ CM_CondInvar,
};
/// Save vectorization decision \p W and \p Cost taken by the cost model for
@@ -1360,6 +1365,10 @@ class LoopVectorizationCostModel {
/// The cost computation for Gather/Scatter instruction.
InstructionCost getGatherScatterCost(Instruction *I, ElementCount VF);
+ /// Cost of lowering a predicated loop-invariant load as a single-lane
+ /// masked load + broadcast. Returns getInvalid() if not applicable.
+ InstructionCost getConditionalInvarCost(Instruction *I, ElementCount VF);
+
/// The cost computation for widening instruction \p I with consecutive
/// memory access.
InstructionCost getConsecutiveMemOpCost(Instruction *I, ElementCount VF,
@@ -2777,9 +2786,11 @@ void LoopVectorizationCostModel::collectLoopUniforms(ElementCount VF) {
if (IsUniformMemOpUse(I))
return true;
+ // CM_CondInvar uses its scalar invariant pointer uniformly.
return (WideningDecision == CM_Widen ||
WideningDecision == CM_Widen_Reverse ||
- WideningDecision == CM_Interleave);
+ WideningDecision == CM_Interleave ||
+ WideningDecision == CM_CondInvar);
};
// Returns true if Ptr is the pointer operand of a memory access instruction
@@ -4379,6 +4390,49 @@ LoopVectorizationCostModel::getGatherScatterCost(Instruction *I,
Config.CostKind);
}
+InstructionCost
+LoopVectorizationCostModel::getConditionalInvarCost(Instruction *I,
+ ElementCount VF) {
+ // Applicability checks. Any failure returns Invalid; the cost ladder
+ // already gates on .isValid().
+ if (!EnableMaskedInvariantLoad)
+ return InstructionCost::getInvalid();
+ if (!Legal->blockNeedsPredication(I->getParent()))
+ return InstructionCost::getInvalid();
+ if (!isMaskRequired(I))
+ return InstructionCost::getInvalid();
+ if (!isa<LoadInst>(I))
+ return InstructionCost::getInvalid();
+ const Align Alignment = getLoadStoreAlignment(I);
+ unsigned AS = getLoadStoreAddressSpace(I);
+ if (!TTI.isLegalMaskedLoad(I->getType(), Alignment, AS))
+ return InstructionCost::getInvalid();
+ Value *Ptr = getLoadStorePointerOperand(I);
+ if (!Legal->isInvariant(Ptr))
+ return InstructionCost::getInvalid();
+
+ // Components: AnyOf or-reduce + lane-0 mask materialisation
+ // (zero-splat + insertelement) + masked load + lane-0 broadcast shuffle.
+ // InstructionCost arithmetic propagates Invalid across the chain.
+ Type *ValTy = getLoadStoreType(I);
+ auto *VectorTy = cast<VectorType>(toVectorTy(ValTy, VF));
+ auto *MaskTy = cast<VectorType>(
+ toVectorTy(Type::getInt1Ty(I->getContext()), VF));
+ InstructionCost Cost = TTI.getArithmeticReductionCost(
+ Instruction::Or, MaskTy, /*FMF*/ std::nullopt, Config.CostKind);
+ Cost += TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, MaskTy, MaskTy,
+ {}, Config.CostKind, 0);
+ Cost += TTI.getVectorInstrCost(Instruction::InsertElement, MaskTy,
+ Config.CostKind, /*Index=*/0);
+ Cost += TTI.getMemIntrinsicInstrCost(
+ MemIntrinsicCostAttributes(Intrinsic::masked_load, VectorTy, Alignment,
+ AS),
+ Config.CostKind);
+ Cost += TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VectorTy,
+ VectorTy, {}, Config.CostKind, 0);
+ return Cost;
+}
+
InstructionCost
LoopVectorizationCostModel::getInterleaveGroupCost(Instruction *I,
ElementCount VF) {
@@ -4782,6 +4836,7 @@ void LoopVectorizationCostModel::setCostBasedWideningDecision(ElementCount VF) {
InstructionCost ScalarizationCost =
getMemInstScalarizationCost(&I, VF) * NumAccesses;
+ InstructionCost ConditionalInvarCost = getConditionalInvarCost(&I, VF);
// Choose better solution for the current VF,
// write down this decision and use it during vectorization.
InstructionCost Cost;
@@ -4790,6 +4845,11 @@ void LoopVectorizationCostModel::setCostBasedWideningDecision(ElementCount VF) {
InterleaveCost < ScalarizationCost) {
Decision = CM_Interleave;
Cost = InterleaveCost;
+ } else if (ConditionalInvarCost.isValid() &&
+ ConditionalInvarCost <= GatherScatterCost &&
+ ConditionalInvarCost < ScalarizationCost) {
+ Decision = CM_CondInvar;
+ Cost = ConditionalInvarCost;
} else if (GatherScatterCost < ScalarizationCost) {
Decision = CM_GatherScatter;
Cost = GatherScatterCost;
@@ -5281,6 +5341,9 @@ LoopVectorizationCostModel::getInstructionCost(Instruction *I,
: TTI::CastContextHint::Normal;
case LoopVectorizationCostModel::CM_Widen_Reverse:
return TTI::CastContextHint::Reversed;
+ case LoopVectorizationCostModel::CM_CondInvar:
+ return isPredicatedInst(I) ? TTI::CastContextHint::Masked
+ : TTI::CastContextHint::Normal;
case LoopVectorizationCostModel::CM_Unknown:
llvm_unreachable("Instr did not go through cost modelling?");
case LoopVectorizationCostModel::CM_InvalidatedDecision:
@@ -6013,6 +6076,7 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
if (EpilogueVecKind == EpilogueVectorizationKind::None)
VPlanTransforms::expandSCEVsToVPInstructions(BestVPlan, *PSE.getSE());
VPlanTransforms::cse(BestVPlan);
+ VPlanTransforms::cseUniformMemoryReads(BestVPlan);
VPlanTransforms::simplifyRecipes(BestVPlan);
// Removing branches and incoming values may expose additional simplification
// opportunities.
@@ -6176,6 +6240,41 @@ bool VPRecipeBuilder::prefersVectorizedAddressing() const {
return CM.TTI.prefersVectorizedAddressing();
}
+VPInstruction *
+VPRecipeBuilder::tryToWidenCondInvarLoad(LoadInst *Load, VPValue *Ptr,
+ VPValue *Mask, VPInstruction *VPI) {
+ assert(Mask && "CM_CondInvar requires a non-null block mask");
+ LLVMContext &Ctx = Load->getContext();
+ DebugLoc DL = Load->getDebugLoc();
+ Builder.setInsertPoint(VPI);
+
+ // any = AnyOf(BlockMask). Unroll appends per-part masks, so this becomes
+ // a single OR-reduction across all UF parts.
+ VPValue *AnyActive = Builder.createNaryOp(VPInstruction::AnyOf, {Mask}, DL);
+
+ // vmsk = insertelement <VF x i1> zeroinitializer, any, 0
+ Type *I1Ty = Type::getInt1Ty(Ctx);
+ VPValue *FalseI1 = Plan.getOrAddLiveIn(ConstantInt::getFalse(I1Ty));
+ VPValue *ZeroVMask =
+ Builder.createNaryOp(VPInstruction::Broadcast, {FalseI1}, DL);
+ VPValue *Lane0Idx =
+ Plan.getOrAddLiveIn(ConstantInt::get(Type::getInt32Ty(Ctx), 0));
+ VPValue *VMask = Builder.createNaryOp(
+ Instruction::InsertElement, {ZeroVMask, AnyActive, Lane0Idx}, DL);
+
+ // Consecutive=true selects the masked-load (not masked-gather) lowering
+ // in VPWidenLoadRecipe::execute. The access is stride-0, not stride-1;
+ // promote to a real AddrKind enum if other passes start relying on
+ // isConsecutive() distinguishing the two.
+ auto *MaskedLoad =
+ new VPWidenLoadRecipe(*Load, Ptr, VMask, /*Consecutive*/ true, *VPI, DL);
+ Builder.insert(MaskedLoad);
+
+ return new VPInstruction(VPInstruction::BroadcastLane,
+ {Lane0Idx, MaskedLoad->getVPSingleValue()},
+ /*Flags*/ {}, /*MD*/ {}, DL);
+}
+
VPRecipeBase *VPRecipeBuilder::tryToWidenMemory(VPInstruction *VPI,
VFRange &Range) {
assert((VPI->getOpcode() == Instruction::Load ||
@@ -6224,6 +6323,8 @@ VPRecipeBase *VPRecipeBuilder::tryToWidenMemory(VPInstruction *VPI,
if (VPI->getOpcode() == Instruction::Load) {
auto *Load = cast<LoadInst>(I);
+ if (Decision == LoopVectorizationCostModel::CM_CondInvar && Mask)
+ return tryToWidenCondInvarLoad(Load, Ptr, Mask, VPI);
auto *LoadR = Builder.createWidenLoad(*Load, Ptr, Mask, Consecutive, *VPI,
Load->getDebugLoc());
if (Reverse)
diff --git a/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h b/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
index 1303b62d5faf4..aff4f87995372 100644
--- a/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
+++ b/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
@@ -38,6 +38,11 @@ class VPRecipeBuilder {
/// Range. The function should not be called for memory instructions or calls.
bool shouldWiden(Instruction *I, VFRange &Range) const;
+ /// Lower a CM_CondInvar \p Load to: single-lane masked-load + broadcast.
+ /// \p Mask is the non-null block mask. Returns the BroadcastLane recipe.
+ VPInstruction *tryToWidenCondInvarLoad(LoadInst *Load, VPValue *Ptr,
+ VPValue *Mask, VPInstruction *VPI);
+
/// Optimize the special case where the operand of \p VPI is a constant
/// integer induction variable.
VPWidenIntOrFpInductionRecipe *
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index 4cb214ac0a257..5b9768dcce66e 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -1317,6 +1317,9 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
LastActiveLane,
// Returns a reversed vector for the operand.
Reverse,
+ // Broadcast lane N of operand 1 across all VF lanes. Operand 0 is a
+ // constant i32 lane index. Operand order matches ExtractLane.
+ BroadcastLane,
/// Start vector for reductions with 3 operands: the original start value,
/// the identity value for the reduction and an integer indicating the
/// scaling factor.
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 7d341cc12ffbe..146dbc3121e54 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -655,6 +655,7 @@ unsigned VPInstruction::getNumOperandsForOpcode() const {
case Instruction::Store:
case VPInstruction::BranchOnCount:
case VPInstruction::BranchOnTwoConds:
+ case VPInstruction::BroadcastLane:
case VPInstruction::FirstOrderRecurrenceSplice:
case VPInstruction::LogicalAnd:
case VPInstruction::LogicalOr:
@@ -731,6 +732,38 @@ static Instruction::BinaryOps getSubRecurOpcode(RecurKind Kind) {
llvm_unreachable("RecurKind should be Sub/FSub.");
}
+/// Broadcast lane \p LaneNo of \p Vec across the result vector.
+/// - Fixed : shufflevector with mask <i32 N, ...>.
+/// - Scalable, lane 0 : shufflevector with zeroinitializer mask.
+/// - Scalable, lane k>0 : extractelement + CreateVectorSplat
+/// (non-zero scalable shuffle masks are rejected
+/// by the verifier today; see
+/// ShuffleVectorInst::isValidOperands).
+static Value *generateBroadcastLane(IRBuilderBase &Builder, Value *Vec,
+ unsigned LaneNo) {
+ auto *VecTy = cast<VectorType>(Vec->getType());
+ Value *Poison = PoisonValue::get(Vec->getType());
+
+ if (isa<ScalableVectorType>(VecTy)) {
+ if (LaneNo == 0) {
+ Type *I32Ty = Type::getInt32Ty(VecTy->getContext());
+ auto *MaskTy = VectorType::get(I32Ty, VecTy->getElementCount());
+ return Builder.CreateShuffleVector(Vec, Poison,
+ Constant::getNullValue(MaskTy),
+ "broadcast.lane");
+ }
+ Value *Scalar =
+ Builder.CreateExtractElement(Vec, Builder.getInt32(LaneNo));
+ return Builder.CreateVectorSplat(VecTy->getElementCount(), Scalar,
+ "broadcast.lane");
+ }
+
+ auto *FVT = cast<FixedVectorType>(VecTy);
+ SmallVector<int, 16> ShufMask(FVT->getNumElements(),
+ static_cast<int>(LaneNo));
+ return Builder.CreateShuffleVector(Vec, Poison, ShufMask, "broadcast.lane");
+}
+
Value *VPInstruction::generate(VPTransformState &State) {
IRBuilderBase &Builder = State.Builder;
@@ -1092,6 +1125,12 @@ Value *VPInstruction::generate(VPTransformState &State) {
return State.get(getOperand(0), true);
case VPInstruction::Reverse:
return Builder.CreateVectorReverse(State.get(getOperand(0)), "reverse");
+ case VPInstruction::BroadcastLane: {
+ Value *Idx = State.get(getOperand(0), /*IsScalar*/ true);
+ Value *Vec = State.get(getOperand(1));
+ unsigned LaneNo = cast<ConstantInt>(Idx)->getZExtValue();
+ return generateBroadcastLane(Builder, Vec, LaneNo);
+ }
case VPInstruction::ExtractLastActive: {
Value *Result = State.get(getOperand(0), /*IsScalar=*/true);
for (unsigned Idx = 1; Idx < getNumOperands(); Idx += 2) {
@@ -1445,6 +1484,19 @@ InstructionCost VPInstruction::computeCost(ElementCount VF,
VectorTy, /*Mask=*/{}, Ctx.CostKind,
/*Index=*/0);
}
+ case VPInstruction::BroadcastLane: {
+ assert(VF.isVector() && "BroadcastLane requires a vector VF");
+ Type *EltTy = getOperand(1)->getScalarType();
+ auto *VectorTy = cast<VectorType>(toVectorTy(EltTy, VF));
+ // Forward the constant lane index so targets can specialise lane-0.
+ unsigned LaneIdx = 0;
+ if (auto *LiveInIdx = getOperand(0)->getLiveInIRValue())
+ if (auto *CI = dyn_cast<ConstantInt>(LiveInIdx))
+ LaneIdx = CI->getZExtValue();
+ return Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VectorTy,
+ VectorTy, /*Mask=*/{}, Ctx.CostKind,
+ /*Index=*/LaneIdx);
+ }
case VPInstruction::ExtractLastLane: {
// Add on the cost of extracting the element.
auto *VecTy = toVectorTy(getOperand(0)->getScalarType(), VF);
@@ -1707,6 +1759,7 @@ bool VPInstruction::usesFirstLaneOnly(const VPValue *Op) const {
return false;
case VPInstruction::ExitingIVValue:
case VPInstruction::ExtractLane:
+ case VPInstruction::BroadcastLane:
return Op == getOperand(0);
};
llvm_unreachable("switch should return");
@@ -1839,6 +1892,9 @@ void VPInstruction::printRecipe(raw_ostream &O, const Twine &Indent,
case VPInstruction::Reverse:
O << "reverse";
break;
+ case VPInstruction::BroadcastLane:
+ O << "broadcast-lane";
+ break;
case VPInstruction::Unpack:
O << "unpack";
break;
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index 3311a9711cb3f..c1ca707e8a1c8 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -2304,6 +2304,42 @@ struct VPCSEDenseMapInfo : public DenseMapInfo<VPSingleDefRecipe *> {
};
} // end anonymous namespace
+/// CSE duplicate masked VPWidenLoadRecipes (which are not VPSingleDefRecipes
+/// and so are missed by the generic cse()). Reset on any write within the
+/// block; unmasked loads are left to downstream EarlyCSE/GVN.
+void VPlanTransforms::cseUniformMemoryReads(VPlan &Plan) {
+ ReversePostOrderTraversal<VPBlockDeepTraversalWrapper<VPBlockBase *>> RPOT(
+ Plan.getEntry());
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(RPOT)) {
+ SmallVector<VPWidenLoadRecipe *, 4> Candidates;
+ for (VPRecipeBase &R : make_early_inc_range(*VPBB)) {
+ if (R.mayWriteToMemory()) {
+ Candidates.clear();
+ continue;
+ }
+ auto *Load = dyn_cast<VPWidenLoadRecipe>(&R);
+ if (!Load || !Load->isMasked())
+ continue;
+ VPWidenLoadRecipe *Match = nullptr;
+ for (VPWidenLoadRecipe *C : Candidates) {
+ if (!equal(C->operands(), Load->operands()))
+ continue;
+ // Same ingredient => same alignment/metadata/type.
+ if (&C->getIngredient() != &Load->getIngredient())
+ continue;
+ Match = C;
+ break;
+ }
+ if (Match) {
+ Load->getVPSingleValue()->replaceAllUsesWith(Match->getVPSingleValue());
+ Load->eraseFromParent();
+ continue;
+ }
+ Candidates.push_back(Load);
+ }
+ }
+}
+
/// Perform a common-subexpression-elimination of VPSingleDefRecipes on the \p
/// Plan.
void VPlanTransforms::cse(VPlan &Plan) {
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index 4b59d37150ff4..c504b381c93e9 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -413,6 +413,10 @@ struct VPlanTransforms {
/// Perform common-subexpression-elimination on \p Plan.
static void cse(VPlan &Plan);
+ /// CSE masked VPWidenLoadRecipes with identical operands inside a basic
+ /// block, ignoring candidates across an intervening write. No AA needed.
+ static void cseUniformMemoryReads(VPlan &Plan);
+
/// If there's a single exit block, optimize its phi recipes that use exiting
/// IV values by feeding them precomputed end values instead, possibly taken
/// one step backwards.
diff --git a/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp b/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp
index a4d6cc0ecb480..b50faf9b0b5ea 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp
@@ -40,6 +40,10 @@ class VPlanVerifier {
/// Verify that \p LastActiveLane's operand is guaranteed to be a prefix-mask.
bool verifyLastActiveLaneRecipe(const VPInstruction &LastActiveLane) const;
+ /// Operand 0 of \p BroadcastLane must be a live-in constant integer
+ /// (shufflevector masks are constant-only).
+ bool verifyBroadcastLaneRecipe(const VPInstruction &BroadcastLane) const;
+
bool verifyVPBasicBlock(const VPBasicBlock *VPBB);
bool verifyBlock(const VPBlockBase *VPB);
@@ -224,6 +228,28 @@ bool VPlanVerifier::verifyLastActiveLaneRecipe(
return true;
}
+bool VPlanVerifier::verifyBroadcastLaneRecipe(
+ const VPInstruction &BroadcastLane) const {
+ assert(BroadcastLane.getOpcode() == VPInstruction::BroadcastLane &&
+ "must be called with VPInstruction::BroadcastLane");
+
+ if (BroadcastLane.getNumOperands() != 2) {
+ errs() << "BroadcastLane must have exactly two operands "
+ "(lane index, vector)\n";
+ return false;
+ }
+
+ const VPValue *LaneIdx = BroadcastLane.getOperand(0);
+ Value *LiveInIdx = LaneIdx->getLiveInIRValue();
+ if (!LiveInIdx || !isa<ConstantInt>(LiveInIdx)) {
+ errs() << "BroadcastLane operand 0 must be a constant integer lane "
+ "index (live-in ConstantInt)\n";
+ return false;
+ }
+
+ return true;
+}
+
bool VPlanVerifier::verifyVPBasicBlock(const VPBasicBlock *VPBB) {
if (!verifyPhiRecipes(VPBB))
return false;
@@ -330,6 +356,10 @@ bool VPlanVerifier::verifyVPBasicBlock(const VPBasicBlock *VPBB) {
if (!verifyLastActiveLaneRecipe(*VPI))
return false;
break;
+ case VPInstruction::BroadcastLane:
+ if (!verifyBroadcastLaneRecipe(*VPI))
+ return false;
+ break;
default:
break;
}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll
new file mode 100644
index 0000000000000..43be17e82ec54
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll
@@ -0,0 +1,93 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=loop-vectorize -enable-masked-invariant-load -force-vector-width=4 \
+; RUN: -force-vector-interleave=1 -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S \
+; RUN: | FileCheck %s
+
+;; Two distinct loop-invariant pointers loaded under the same predicate in the
+;; same block. Each load is independently lowered to a single-lane masked load
+;; + broadcast, but they share the SAME AnyOf(mask) and lane-0 mask material
+;; thanks to the standard VPlan CSE that runs before unrolling.
+;;
+;; for (i = 0; i < n; ++i)
+;; if (cond[i])
+;; a[i] = *inv1 + *inv2;
+
+define void @cond_invar_load_two_ptrs(ptr noalias %a, ptr noalias %cond, ptr noalias %inv1, ptr noalias %inv2, i64 %n) {
+; CHECK-LABEL: @cond_invar_load_two_ptrs(
+; CHECK-NEXT: entry:
+; CHECK-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N:%.*]], 4
+; CHECK-NEXT: br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK: vector.ph:
+; CHECK-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; CHECK-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
+; CHECK: vector.body:
+; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
+; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; CHECK-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; CHECK-NEXT: [[TMP2:%.*]] = freeze <4 x i1> [[TMP1]]
+; CHECK-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; CHECK-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
+; CHECK-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV1:%.*]], <4 x i1> [[TMP4]], <4 x i32> poison)
+; CHECK-NEXT: [[BROADCAST_LANE:%.*]] = shufflevector <4 x i32> [[WIDE_MASKED_LOAD]], <4 x i32> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT: [[WIDE_MASKED_LOAD1:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV2:%.*]], <4 x i1> [[TMP4]], <4 x i32> poison)
+; CHECK-NEXT: [[BROADCAST_LANE2:%.*]] = shufflevector <4 x i32> [[WIDE_MASKED_LOAD1]], <4 x i32> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT: [[TMP5:%.*]] = add <4 x i32> [[BROADCAST_LANE]], [[BROADCAST_LANE2]]
+; CHECK-NEXT: [[TMP6:%.*]] = getelementptr i32, ptr [[A:%.*]], i64 [[INDEX]]
+; CHECK-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[TMP5]], ptr align 4 [[TMP6]], <4 x i1> [[TMP1]])
+; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK: middle.block:
+; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
+; CHECK: scalar.ph:
+; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-NEXT: br label [[FOR_BODY:%.*]]
+; CHECK: for.body:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[I_NEXT:%.*]], [[FOR_INC:%.*]] ]
+; CHECK-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; CHECK-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; CHECK-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; CHECK-NEXT: br i1 [[TOBOOL]], label [[IF_THEN:%.*]], label [[FOR_INC]]
+; CHECK: if.then:
+; CHECK-NEXT: [[V1:%.*]] = load i32, ptr [[INV1]], align 4
+; CHECK-NEXT: [[V2:%.*]] = load i32, ptr [[INV2]], align 4
+; CHECK-NEXT: [[SUM:%.*]] = add i32 [[V1]], [[V2]]
+; CHECK-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store i32 [[SUM]], ptr [[A_GEP]], align 4
+; CHECK-NEXT: br label [[FOR_INC]]
+; CHECK: for.inc:
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; CHECK-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; CHECK: for.end:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
+ %cv = load i32, ptr %cond.gep, align 4
+ %tobool = icmp ne i32 %cv, 0
+ br i1 %tobool, label %if.then, label %for.inc
+
+if.then:
+ %v1 = load i32, ptr %inv1, align 4
+ %v2 = load i32, ptr %inv2, align 4
+ %sum = add i32 %v1, %v2
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %sum, ptr %a.gep, align 4
+ br label %for.inc
+
+for.inc:
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll
new file mode 100644
index 0000000000000..e502761b8a7e2
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll
@@ -0,0 +1,64 @@
+; RUN: opt < %s -passes=loop-vectorize -enable-masked-invariant-load -force-vector-width=4 \
+; RUN: -force-vector-interleave=1 -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S \
+; RUN: | FileCheck %s
+
+;; Negative test for CM_CondInvar.
+;;
+;; The conditional load's pointer (%src + i) is loop-VARIANT, so even with
+;; -enable-masked-invariant-load enabled, getConditionalInvarCost() returns
+;; Invalid and the cost model picks the regular consecutive masked load
+;; (CM_Widen).
+;;
+;; for (i = 0; i < n; ++i)
+;; if (cond[i])
+;; a[i] = src[i];
+;;
+;; The output:
+;; * Must contain a regular widened consecutive masked load on src + index.
+;; * Must NOT contain any AnyOf-style i1 OR-reduction.
+;; * Must NOT contain a broadcast.lane shuffle (the CondInvar lowering
+;; marker).
+;; * Must NOT contain a masked.load whose pointer is the loop-invariant
+;; %src argument.
+
+define void @cond_load_not_invariant(ptr noalias %a, ptr noalias %src, ptr noalias %cond, i64 %n) {
+; CHECK-LABEL: define void @cond_load_not_invariant(
+
+; The vector body widens the consecutive masked load on src[index].
+; CHECK: vector.body:
+; CHECK: [[SRC_GEP_V:%.*]] = getelementptr i32, ptr %src, i64 {{%.*}}
+; CHECK: call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[SRC_GEP_V]],
+
+; The CondInvar lowering must NOT be applied.
+; CHECK-NOT: call i1 @llvm.vector.reduce.or
+; CHECK-NOT: %broadcast.lane = shufflevector
+; CHECK-NOT: call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 %src,
+
+; CHECK: middle.block:
+; CHECK: ret void
+
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
+ %cv = load i32, ptr %cond.gep, align 4
+ %tobool = icmp ne i32 %cv, 0
+ br i1 %tobool, label %if.then, label %for.inc
+
+if.then:
+ %src.gep = getelementptr inbounds i32, ptr %src, i64 %i
+ %v = load i32, ptr %src.gep, align 4
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %v, ptr %a.gep, align 4
+ br label %for.inc
+
+for.inc:
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll
new file mode 100644
index 0000000000000..4a66d907675d0
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll
@@ -0,0 +1,169 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
+; RUN: -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S | FileCheck %s --check-prefix=CHECK-OFF
+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
+; RUN: -enable-masked-invariant-load -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S | FileCheck %s --check-prefix=CHECK-ON
+
+;; Conditional load from a loop-invariant pointer.
+;;
+;; for (i = 0; i < n; ++i)
+;; if (cond[i])
+;; a[i] = *inv;
+;;
+;; The pointer %inv is loop-invariant and the load is predicated by %tobool.
+;; Without -enable-masked-invariant-load the cost model falls back to per-lane
+;; scalarised predicated loads (CM_Scalarize) - one pred.load.if/continue
+;; chain per VF lane.
+;;
+;; With -enable-masked-invariant-load the load becomes:
+;; * AnyOf reduction across the block mask,
+;; * lane-0 mask materialisation (zeroinitializer + insertelement i1),
+;; * one masked.load from %inv,
+;; * shufflevector broadcast of lane 0 to all VF lanes.
+
+define void @cond_invar_load(ptr noalias %a, ptr noalias %cond, ptr noalias %inv, i64 %n) {
+; CHECK-OFF-LABEL: @cond_invar_load(
+; CHECK-OFF-NEXT: entry:
+; CHECK-OFF-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N:%.*]], 4
+; CHECK-OFF-NEXT: br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK-OFF: vector.ph:
+; CHECK-OFF-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; CHECK-OFF-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; CHECK-OFF-NEXT: br label [[VECTOR_BODY:%.*]]
+; CHECK-OFF: vector.body:
+; CHECK-OFF-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
+; CHECK-OFF-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
+; CHECK-OFF-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; CHECK-OFF-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; CHECK-OFF-NEXT: [[TMP2:%.*]] = extractelement <4 x i1> [[TMP1]], i64 0
+; CHECK-OFF-NEXT: br i1 [[TMP2]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
+; CHECK-OFF: pred.load.if:
+; CHECK-OFF-NEXT: [[TMP3:%.*]] = load i32, ptr [[INV:%.*]], align 4
+; CHECK-OFF-NEXT: [[TMP4:%.*]] = insertelement <4 x i32> poison, i32 [[TMP3]], i64 0
+; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE]]
+; CHECK-OFF: pred.load.continue:
+; CHECK-OFF-NEXT: [[TMP5:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP4]], [[PRED_LOAD_IF]] ]
+; CHECK-OFF-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP1]], i64 1
+; CHECK-OFF-NEXT: br i1 [[TMP6]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
+; CHECK-OFF: pred.load.if1:
+; CHECK-OFF-NEXT: [[TMP7:%.*]] = load i32, ptr [[INV]], align 4
+; CHECK-OFF-NEXT: [[TMP8:%.*]] = insertelement <4 x i32> [[TMP5]], i32 [[TMP7]], i64 1
+; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE2]]
+; CHECK-OFF: pred.load.continue2:
+; CHECK-OFF-NEXT: [[TMP9:%.*]] = phi <4 x i32> [ [[TMP5]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP8]], [[PRED_LOAD_IF1]] ]
+; CHECK-OFF-NEXT: [[TMP10:%.*]] = extractelement <4 x i1> [[TMP1]], i64 2
+; CHECK-OFF-NEXT: br i1 [[TMP10]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
+; CHECK-OFF: pred.load.if3:
+; CHECK-OFF-NEXT: [[TMP11:%.*]] = load i32, ptr [[INV]], align 4
+; CHECK-OFF-NEXT: [[TMP12:%.*]] = insertelement <4 x i32> [[TMP9]], i32 [[TMP11]], i64 2
+; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE4]]
+; CHECK-OFF: pred.load.continue4:
+; CHECK-OFF-NEXT: [[TMP13:%.*]] = phi <4 x i32> [ [[TMP9]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP12]], [[PRED_LOAD_IF3]] ]
+; CHECK-OFF-NEXT: [[TMP14:%.*]] = extractelement <4 x i1> [[TMP1]], i64 3
+; CHECK-OFF-NEXT: br i1 [[TMP14]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6]]
+; CHECK-OFF: pred.load.if5:
+; CHECK-OFF-NEXT: [[TMP15:%.*]] = load i32, ptr [[INV]], align 4
+; CHECK-OFF-NEXT: [[TMP16:%.*]] = insertelement <4 x i32> [[TMP13]], i32 [[TMP15]], i64 3
+; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE6]]
+; CHECK-OFF: pred.load.continue6:
+; CHECK-OFF-NEXT: [[TMP17:%.*]] = phi <4 x i32> [ [[TMP13]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP16]], [[PRED_LOAD_IF5]] ]
+; CHECK-OFF-NEXT: [[TMP18:%.*]] = getelementptr i32, ptr [[A:%.*]], i64 [[INDEX]]
+; CHECK-OFF-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[TMP17]], ptr align 4 [[TMP18]], <4 x i1> [[TMP1]])
+; CHECK-OFF-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-OFF-NEXT: [[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-OFF-NEXT: br i1 [[TMP19]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-OFF: middle.block:
+; CHECK-OFF-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; CHECK-OFF-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
+; CHECK-OFF: scalar.ph:
+; CHECK-OFF-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-OFF-NEXT: br label [[FOR_BODY:%.*]]
+; CHECK-OFF: for.body:
+; CHECK-OFF-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[I_NEXT:%.*]], [[FOR_INC:%.*]] ]
+; CHECK-OFF-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; CHECK-OFF-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; CHECK-OFF-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; CHECK-OFF-NEXT: br i1 [[TOBOOL]], label [[IF_THEN:%.*]], label [[FOR_INC]]
+; CHECK-OFF: if.then:
+; CHECK-OFF-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; CHECK-OFF-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-OFF-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; CHECK-OFF-NEXT: br label [[FOR_INC]]
+; CHECK-OFF: for.inc:
+; CHECK-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; CHECK-OFF-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; CHECK-OFF-NEXT: br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; CHECK-OFF: for.end:
+; CHECK-OFF-NEXT: ret void
+;
+; CHECK-ON-LABEL: @cond_invar_load(
+; CHECK-ON-NEXT: entry:
+; CHECK-ON-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N:%.*]], 4
+; CHECK-ON-NEXT: br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
+; CHECK-ON: vector.ph:
+; CHECK-ON-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; CHECK-ON-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; CHECK-ON-NEXT: br label [[VECTOR_BODY:%.*]]
+; CHECK-ON: vector.body:
+; CHECK-ON-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
+; CHECK-ON-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
+; CHECK-ON-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; CHECK-ON-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; CHECK-ON-NEXT: [[TMP2:%.*]] = freeze <4 x i1> [[TMP1]]
+; CHECK-ON-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; CHECK-ON-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
+; CHECK-ON-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV:%.*]], <4 x i1> [[TMP4]], <4 x i32> poison)
+; CHECK-ON-NEXT: [[BROADCAST_LANE:%.*]] = shufflevector <4 x i32> [[WIDE_MASKED_LOAD]], <4 x i32> poison, <4 x i32> zeroinitializer
+; CHECK-ON-NEXT: [[TMP5:%.*]] = getelementptr i32, ptr [[A:%.*]], i64 [[INDEX]]
+; CHECK-ON-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[BROADCAST_LANE]], ptr align 4 [[TMP5]], <4 x i1> [[TMP1]])
+; CHECK-ON-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-ON-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-ON-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; CHECK-ON: middle.block:
+; CHECK-ON-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; CHECK-ON-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
+; CHECK-ON: scalar.ph:
+; CHECK-ON-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
+; CHECK-ON-NEXT: br label [[FOR_BODY:%.*]]
+; CHECK-ON: for.body:
+; CHECK-ON-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[I_NEXT:%.*]], [[FOR_INC:%.*]] ]
+; CHECK-ON-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; CHECK-ON-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; CHECK-ON-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; CHECK-ON-NEXT: br i1 [[TOBOOL]], label [[IF_THEN:%.*]], label [[FOR_INC]]
+; CHECK-ON: if.then:
+; CHECK-ON-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; CHECK-ON-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-ON-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; CHECK-ON-NEXT: br label [[FOR_INC]]
+; CHECK-ON: for.inc:
+; CHECK-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; CHECK-ON-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; CHECK-ON-NEXT: br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; CHECK-ON: for.end:
+; CHECK-ON-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
+ %cv = load i32, ptr %cond.gep, align 4
+ %tobool = icmp ne i32 %cv, 0
+ br i1 %tobool, label %if.then, label %for.inc
+
+if.then:
+ %v = load i32, ptr %inv, align 4
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %v, ptr %a.gep, align 4
+ br label %for.inc
+
+for.inc:
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
>From b7104c6ae5d0ec72b8f9a672b086990cf2a3dd7a Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Tue, 2 Jun 2026 17:59:32 +0530
Subject: [PATCH 2/5] Replace CM_CondInvar and its recipe-builder dispatch with
a VPlan
transform widenPredicatedInvariantLoads, run after sinkPredicatedStores.
It rewrites masked replicate loads from loop-invariant addresses into
AnyOf(mask) + masked VPWidenLoadRecipe + BroadcastLane(0);
---
.../Transforms/Vectorize/LoopVectorize.cpp | 105 +-----------------
.../Transforms/Vectorize/VPRecipeBuilder.h | 5 -
.../Transforms/Vectorize/VPlanTransforms.cpp | 78 +++++++++++++
.../Transforms/Vectorize/VPlanTransforms.h | 7 ++
4 files changed, 88 insertions(+), 107 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 0084bf0ce3a03..78b5afab80eb4 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -206,10 +206,6 @@ static cl::opt<bool> ForcePartialAliasingVectorization(
"force-partial-aliasing-vectorization", cl::init(false), cl::Hidden,
cl::desc("Replace pointer diff checks with alias masks."));
-static cl::opt<bool> EnableMaskedInvariantLoad(
- "enable-masked-invariant-load", cl::init(false), cl::ReallyHidden,
- cl::desc("Vectorize conditional invariant load with mask-load"
- " and broadcast"));
/// Option tail-folding-policy controls the tail-folding strategy and lists all
/// available options. The vectorizer will attempt to fold the tail-loop into
/// the vector loop (main/epilogue loops) and predicate the instructions
@@ -886,8 +882,7 @@ class LoopVectorizationCostModel {
/// A widening decision that has been invalidated after replacing the
/// corresponding recipe during VPlan transforms.
/// TODO: Remove once the legacy exit cost computation is retired.
- CM_InvalidatedDecision,
- CM_CondInvar,
+ CM_InvalidatedDecision
};
/// Save vectorization decision \p W and \p Cost taken by the cost model for
@@ -1365,10 +1360,6 @@ class LoopVectorizationCostModel {
/// The cost computation for Gather/Scatter instruction.
InstructionCost getGatherScatterCost(Instruction *I, ElementCount VF);
- /// Cost of lowering a predicated loop-invariant load as a single-lane
- /// masked load + broadcast. Returns getInvalid() if not applicable.
- InstructionCost getConditionalInvarCost(Instruction *I, ElementCount VF);
-
/// The cost computation for widening instruction \p I with consecutive
/// memory access.
InstructionCost getConsecutiveMemOpCost(Instruction *I, ElementCount VF,
@@ -2786,11 +2777,9 @@ void LoopVectorizationCostModel::collectLoopUniforms(ElementCount VF) {
if (IsUniformMemOpUse(I))
return true;
- // CM_CondInvar uses its scalar invariant pointer uniformly.
return (WideningDecision == CM_Widen ||
WideningDecision == CM_Widen_Reverse ||
- WideningDecision == CM_Interleave ||
- WideningDecision == CM_CondInvar);
+ WideningDecision == CM_Interleave);
};
// Returns true if Ptr is the pointer operand of a memory access instruction
@@ -4390,49 +4379,6 @@ LoopVectorizationCostModel::getGatherScatterCost(Instruction *I,
Config.CostKind);
}
-InstructionCost
-LoopVectorizationCostModel::getConditionalInvarCost(Instruction *I,
- ElementCount VF) {
- // Applicability checks. Any failure returns Invalid; the cost ladder
- // already gates on .isValid().
- if (!EnableMaskedInvariantLoad)
- return InstructionCost::getInvalid();
- if (!Legal->blockNeedsPredication(I->getParent()))
- return InstructionCost::getInvalid();
- if (!isMaskRequired(I))
- return InstructionCost::getInvalid();
- if (!isa<LoadInst>(I))
- return InstructionCost::getInvalid();
- const Align Alignment = getLoadStoreAlignment(I);
- unsigned AS = getLoadStoreAddressSpace(I);
- if (!TTI.isLegalMaskedLoad(I->getType(), Alignment, AS))
- return InstructionCost::getInvalid();
- Value *Ptr = getLoadStorePointerOperand(I);
- if (!Legal->isInvariant(Ptr))
- return InstructionCost::getInvalid();
-
- // Components: AnyOf or-reduce + lane-0 mask materialisation
- // (zero-splat + insertelement) + masked load + lane-0 broadcast shuffle.
- // InstructionCost arithmetic propagates Invalid across the chain.
- Type *ValTy = getLoadStoreType(I);
- auto *VectorTy = cast<VectorType>(toVectorTy(ValTy, VF));
- auto *MaskTy = cast<VectorType>(
- toVectorTy(Type::getInt1Ty(I->getContext()), VF));
- InstructionCost Cost = TTI.getArithmeticReductionCost(
- Instruction::Or, MaskTy, /*FMF*/ std::nullopt, Config.CostKind);
- Cost += TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, MaskTy, MaskTy,
- {}, Config.CostKind, 0);
- Cost += TTI.getVectorInstrCost(Instruction::InsertElement, MaskTy,
- Config.CostKind, /*Index=*/0);
- Cost += TTI.getMemIntrinsicInstrCost(
- MemIntrinsicCostAttributes(Intrinsic::masked_load, VectorTy, Alignment,
- AS),
- Config.CostKind);
- Cost += TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VectorTy,
- VectorTy, {}, Config.CostKind, 0);
- return Cost;
-}
-
InstructionCost
LoopVectorizationCostModel::getInterleaveGroupCost(Instruction *I,
ElementCount VF) {
@@ -4836,7 +4782,6 @@ void LoopVectorizationCostModel::setCostBasedWideningDecision(ElementCount VF) {
InstructionCost ScalarizationCost =
getMemInstScalarizationCost(&I, VF) * NumAccesses;
- InstructionCost ConditionalInvarCost = getConditionalInvarCost(&I, VF);
// Choose better solution for the current VF,
// write down this decision and use it during vectorization.
InstructionCost Cost;
@@ -4845,11 +4790,6 @@ void LoopVectorizationCostModel::setCostBasedWideningDecision(ElementCount VF) {
InterleaveCost < ScalarizationCost) {
Decision = CM_Interleave;
Cost = InterleaveCost;
- } else if (ConditionalInvarCost.isValid() &&
- ConditionalInvarCost <= GatherScatterCost &&
- ConditionalInvarCost < ScalarizationCost) {
- Decision = CM_CondInvar;
- Cost = ConditionalInvarCost;
} else if (GatherScatterCost < ScalarizationCost) {
Decision = CM_GatherScatter;
Cost = GatherScatterCost;
@@ -5341,9 +5281,6 @@ LoopVectorizationCostModel::getInstructionCost(Instruction *I,
: TTI::CastContextHint::Normal;
case LoopVectorizationCostModel::CM_Widen_Reverse:
return TTI::CastContextHint::Reversed;
- case LoopVectorizationCostModel::CM_CondInvar:
- return isPredicatedInst(I) ? TTI::CastContextHint::Masked
- : TTI::CastContextHint::Normal;
case LoopVectorizationCostModel::CM_Unknown:
llvm_unreachable("Instr did not go through cost modelling?");
case LoopVectorizationCostModel::CM_InvalidatedDecision:
@@ -6240,41 +6177,6 @@ bool VPRecipeBuilder::prefersVectorizedAddressing() const {
return CM.TTI.prefersVectorizedAddressing();
}
-VPInstruction *
-VPRecipeBuilder::tryToWidenCondInvarLoad(LoadInst *Load, VPValue *Ptr,
- VPValue *Mask, VPInstruction *VPI) {
- assert(Mask && "CM_CondInvar requires a non-null block mask");
- LLVMContext &Ctx = Load->getContext();
- DebugLoc DL = Load->getDebugLoc();
- Builder.setInsertPoint(VPI);
-
- // any = AnyOf(BlockMask). Unroll appends per-part masks, so this becomes
- // a single OR-reduction across all UF parts.
- VPValue *AnyActive = Builder.createNaryOp(VPInstruction::AnyOf, {Mask}, DL);
-
- // vmsk = insertelement <VF x i1> zeroinitializer, any, 0
- Type *I1Ty = Type::getInt1Ty(Ctx);
- VPValue *FalseI1 = Plan.getOrAddLiveIn(ConstantInt::getFalse(I1Ty));
- VPValue *ZeroVMask =
- Builder.createNaryOp(VPInstruction::Broadcast, {FalseI1}, DL);
- VPValue *Lane0Idx =
- Plan.getOrAddLiveIn(ConstantInt::get(Type::getInt32Ty(Ctx), 0));
- VPValue *VMask = Builder.createNaryOp(
- Instruction::InsertElement, {ZeroVMask, AnyActive, Lane0Idx}, DL);
-
- // Consecutive=true selects the masked-load (not masked-gather) lowering
- // in VPWidenLoadRecipe::execute. The access is stride-0, not stride-1;
- // promote to a real AddrKind enum if other passes start relying on
- // isConsecutive() distinguishing the two.
- auto *MaskedLoad =
- new VPWidenLoadRecipe(*Load, Ptr, VMask, /*Consecutive*/ true, *VPI, DL);
- Builder.insert(MaskedLoad);
-
- return new VPInstruction(VPInstruction::BroadcastLane,
- {Lane0Idx, MaskedLoad->getVPSingleValue()},
- /*Flags*/ {}, /*MD*/ {}, DL);
-}
-
VPRecipeBase *VPRecipeBuilder::tryToWidenMemory(VPInstruction *VPI,
VFRange &Range) {
assert((VPI->getOpcode() == Instruction::Load ||
@@ -6323,8 +6225,6 @@ VPRecipeBase *VPRecipeBuilder::tryToWidenMemory(VPInstruction *VPI,
if (VPI->getOpcode() == Instruction::Load) {
auto *Load = cast<LoadInst>(I);
- if (Decision == LoopVectorizationCostModel::CM_CondInvar && Mask)
- return tryToWidenCondInvarLoad(Load, Ptr, Mask, VPI);
auto *LoadR = Builder.createWidenLoad(*Load, Ptr, Mask, Consecutive, *VPI,
Load->getDebugLoc());
if (Reverse)
@@ -6717,6 +6617,7 @@ void LoopVectorizationPlanner::buildVPlans(VPlan &VPlan1, ElementCount MinVF,
// Now optimize the initial VPlan.
RUN_VPLAN_PASS(VPlanTransforms::hoistPredicatedLoads, *Plan, PSE, OrigLoop);
RUN_VPLAN_PASS(VPlanTransforms::sinkPredicatedStores, *Plan, PSE, OrigLoop);
+ RUN_VPLAN_PASS(VPlanTransforms::widenPredicatedInvariantLoads, *Plan, TTI);
RUN_VPLAN_PASS(VPlanTransforms::truncateToMinimalBitwidths, *Plan,
Config.getMinimalBitwidths());
RUN_VPLAN_PASS(VPlanTransforms::optimize, *Plan);
diff --git a/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h b/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
index aff4f87995372..1303b62d5faf4 100644
--- a/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
+++ b/llvm/lib/Transforms/Vectorize/VPRecipeBuilder.h
@@ -38,11 +38,6 @@ class VPRecipeBuilder {
/// Range. The function should not be called for memory instructions or calls.
bool shouldWiden(Instruction *I, VFRange &Range) const;
- /// Lower a CM_CondInvar \p Load to: single-lane masked-load + broadcast.
- /// \p Mask is the non-null block mask. Returns the BroadcastLane recipe.
- VPInstruction *tryToWidenCondInvarLoad(LoadInst *Load, VPValue *Ptr,
- VPValue *Mask, VPInstruction *VPI);
-
/// Optimize the special case where the operand of \p VPI is a constant
/// integer induction variable.
VPWidenIntOrFpInductionRecipe *
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index c1ca707e8a1c8..0f58276040715 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -3883,6 +3883,84 @@ void VPlanTransforms::hoistPredicatedLoads(VPlan &Plan,
}
}
+static cl::opt<bool> EnableMaskedInvariantLoad(
+ "enable-masked-invariant-load", cl::init(false), cl::ReallyHidden,
+ cl::desc("Rewrite predicated loads from loop-invariant addresses as a "
+ "single-lane masked load + broadcast"));
+
+void VPlanTransforms::widenPredicatedInvariantLoads(
+ VPlan &Plan, const TargetTransformInfo &TTI) {
+ if (!EnableMaskedInvariantLoad)
+ return;
+
+ VPRegionBlock *LoopRegion = Plan.getVectorLoopRegion();
+ if (!LoopRegion)
+ return;
+
+ // Collect masked, vector-producing replicate loads whose pointer is defined
+ // outside the loop region and which the target can lower as a masked load.
+ SmallVector<VPReplicateRecipe *> Worklist;
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_shallow(LoopRegion->getEntry()))) {
+ for (VPRecipeBase &R : *VPBB) {
+ auto *RepR = dyn_cast<VPReplicateRecipe>(&R);
+ if (!RepR || !RepR->isPredicated() || RepR->isSingleScalar())
+ continue;
+ if (RepR->getOpcode() != Instruction::Load)
+ continue;
+ VPValue *Addr = RepR->getOperand(0);
+ if (!Addr->isDefinedOutsideLoopRegions())
+ continue;
+ auto *LI = cast<LoadInst>(RepR->getUnderlyingInstr());
+ if (!TTI.isLegalMaskedLoad(LI->getType(), LI->getAlign(),
+ LI->getPointerAddressSpace()))
+ continue;
+ Worklist.push_back(RepR);
+ }
+ }
+
+ for (VPReplicateRecipe *RepR : Worklist) {
+ auto *LI = cast<LoadInst>(RepR->getUnderlyingInstr());
+ LLVMContext &Ctx = LI->getContext();
+ DebugLoc DL = LI->getDebugLoc();
+ VPValue *Addr = RepR->getOperand(0);
+ VPValue *Mask = RepR->getMask();
+
+ VPBuilder Builder(RepR);
+
+ // any = AnyOf(mask). Unroll appends per-part masks, so this becomes a
+ // single OR-reduction across all UF parts.
+ VPValue *AnyActive =
+ Builder.createNaryOp(VPInstruction::AnyOf, {Mask}, DL);
+
+ // vmsk = insertelement <VF x i1> zeroinitializer, any, 0
+ Type *I1Ty = Type::getInt1Ty(Ctx);
+ VPValue *FalseI1 = Plan.getOrAddLiveIn(ConstantInt::getFalse(I1Ty));
+ VPValue *ZeroVMask =
+ Builder.createNaryOp(VPInstruction::Broadcast, {FalseI1}, DL);
+ VPValue *Lane0Idx =
+ Plan.getOrAddLiveIn(ConstantInt::get(Type::getInt32Ty(Ctx), 0));
+ VPValue *VMask = Builder.createNaryOp(
+ Instruction::InsertElement, {ZeroVMask, AnyActive, Lane0Idx}, DL);
+
+ // Consecutive=true selects the masked-load (not masked-gather) lowering in
+ // VPWidenLoadRecipe::execute. The access is stride-0, not stride-1; the
+ // BroadcastLane below splats lane 0 across all VF lanes.
+ auto *MaskedLoad = new VPWidenLoadRecipe(*LI, Addr, VMask,
+ /*Consecutive*/ true, *RepR, DL);
+ Builder.insert(MaskedLoad);
+
+ auto *Bcast = new VPInstruction(
+ VPInstruction::BroadcastLane,
+ {Lane0Idx, MaskedLoad->getVPSingleValue()}, /*Flags*/ {},
+ /*MD*/ {}, DL);
+ Builder.insert(Bcast);
+
+ RepR->replaceAllUsesWith(Bcast);
+ RepR->eraseFromParent();
+ }
+}
+
static bool
canSinkStoreWithNoAliasCheck(ArrayRef<VPReplicateRecipe *> StoresToSink,
PredicatedScalarEvolution &PSE, const Loop &L) {
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index c504b381c93e9..ba1c6e7428a9b 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -433,6 +433,13 @@ struct VPlanTransforms {
static void hoistPredicatedLoads(VPlan &Plan, PredicatedScalarEvolution &PSE,
const Loop *L);
+ /// Rewrite predicated VPReplicateRecipe loads from a loop-invariant address
+ /// into a single-lane masked load + broadcast pattern. Only applied when
+ /// the target reports isLegalMaskedLoad for the element type. Gated by
+ /// -enable-masked-invariant-load.
+ static void widenPredicatedInvariantLoads(VPlan &Plan,
+ const TargetTransformInfo &TTI);
+
/// Sink predicated stores to the same address with complementary predicates
/// (P and NOT P) to an unconditional store with select recipes for the
/// stored values. This eliminates branching overhead when all paths
>From 7ad8581b949e65a21f4ebfc33df0d875ed3dda00 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Tue, 28 Jul 2026 22:43:58 +0530
Subject: [PATCH 3/5] [LV] Address review comments for
widenPredicatedInvariantLoads
- Remove the BroadcastLane VPInstruction; build the broadcast from the
existing ExtractElement + Broadcast recipes instead.
- Gate the rewrite on cost via static recipe cost helpers, lowering only
when it is no costlier than the scalarized load for every vector VF.
- Drop cseUniformMemoryReads (to be handled separately).
- Consolidate the X86 tests into target-independent ones and add
cost/legality gating coverage.
---
.../Vectorize/LoopVectorizationPlanner.cpp | 3 +-
.../Transforms/Vectorize/LoopVectorize.cpp | 5 +-
llvm/lib/Transforms/Vectorize/VPlan.h | 27 +-
.../lib/Transforms/Vectorize/VPlanRecipes.cpp | 95 +---
.../Transforms/Vectorize/VPlanTransforms.cpp | 98 ++--
.../Transforms/Vectorize/VPlanTransforms.h | 12 +-
.../Transforms/Vectorize/VPlanVerifier.cpp | 30 -
.../enable-masked-invariant-load-multiple.ll | 93 ----
...ble-masked-invariant-load-not-invariant.ll | 64 ---
.../X86/enable-masked-invariant-load.ll | 169 ------
.../masked-invariant-load-gate.ll | 254 +++++++++
.../LoopVectorize/masked-invariant-load.ll | 526 ++++++++++++++++++
12 files changed, 889 insertions(+), 487 deletions(-)
delete mode 100644 llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll
delete mode 100644 llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll
delete mode 100644 llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/masked-invariant-load-gate.ll
create mode 100644 llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
index fd7fd2e011a83..bb62e864d6b4c 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.cpp
@@ -13,6 +13,7 @@
//===----------------------------------------------------------------------===//
#include "LoopVectorizationPlanner.h"
+#include "VPlanTransforms.h"
#include "llvm/Analysis/LoopInfo.h"
#include "llvm/Analysis/OptimizationRemarkEmitter.h"
#include "llvm/Analysis/ScalarEvolution.h"
@@ -57,7 +58,7 @@ cl::opt<bool> llvm::PreferInLoopReductions(
/// Note: This currently only applies to `llvm.masked.load` and
/// `llvm.masked.store`. TODO: Extend this to cover other operations as needed.
-static cl::opt<bool> ForceTargetSupportsMaskedMemoryOps(
+cl::opt<bool> llvm::ForceTargetSupportsMaskedMemoryOps(
"force-target-supports-masked-memory-ops", cl::init(false), cl::Hidden,
cl::desc("Assume the target supports masked memory operations (used for "
"testing)."));
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 78b5afab80eb4..e92c4b900bca3 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -6013,7 +6013,6 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
if (EpilogueVecKind == EpilogueVectorizationKind::None)
VPlanTransforms::expandSCEVsToVPInstructions(BestVPlan, *PSE.getSE());
VPlanTransforms::cse(BestVPlan);
- VPlanTransforms::cseUniformMemoryReads(BestVPlan);
VPlanTransforms::simplifyRecipes(BestVPlan);
// Removing branches and incoming values may expose additional simplification
// opportunities.
@@ -6617,7 +6616,9 @@ void LoopVectorizationPlanner::buildVPlans(VPlan &VPlan1, ElementCount MinVF,
// Now optimize the initial VPlan.
RUN_VPLAN_PASS(VPlanTransforms::hoistPredicatedLoads, *Plan, PSE, OrigLoop);
RUN_VPLAN_PASS(VPlanTransforms::sinkPredicatedStores, *Plan, PSE, OrigLoop);
- RUN_VPLAN_PASS(VPlanTransforms::widenPredicatedInvariantLoads, *Plan, TTI);
+ VPCostContext InvarLoadCostCtx(*CM.TLI, *Plan, CM, Config);
+ RUN_VPLAN_PASS(VPlanTransforms::widenPredicatedInvariantLoads, *Plan,
+ InvarLoadCostCtx);
RUN_VPLAN_PASS(VPlanTransforms::truncateToMinimalBitwidths, *Plan,
Config.getMinimalBitwidths());
RUN_VPLAN_PASS(VPlanTransforms::optimize, *Plan);
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index 5b9768dcce66e..a5fa558002c4e 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -1317,9 +1317,6 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
LastActiveLane,
// Returns a reversed vector for the operand.
Reverse,
- // Broadcast lane N of operand 1 across all VF lanes. Operand 0 is a
- // constant i32 lane index. Operand order matches ExtractLane.
- BroadcastLane,
/// Start vector for reductions with 3 operands: the original start value,
/// the identity value for the reduction and an integer indicating the
/// scaling factor.
@@ -1436,6 +1433,18 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
InstructionCost computeCost(ElementCount VF,
VPCostContext &Ctx) const override;
+ /// Return the cost of an AnyOf (OR-reduction) over the mask vector type
+ /// \p MaskTy. Static helper so callers can price the operation without
+ /// constructing a recipe.
+ static InstructionCost computeAnyOfCost(VectorType *MaskTy,
+ VPCostContext &Ctx);
+
+ /// Return the cost of extracting a single element from vector type \p VecTy.
+ /// Static helper so callers can price the operation without constructing a
+ /// recipe.
+ static InstructionCost computeExtractElementCost(VectorType *VecTy,
+ VPCostContext &Ctx);
+
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
/// Print the VPInstruction to dbgs() (for debugging).
LLVM_DUMP_METHOD void dump() const;
@@ -3483,6 +3492,10 @@ class LLVM_ABI_FOR_TEST VPReplicateRecipe : public VPRecipeWithIRFlags,
unsigned getOpcode() const { return getUnderlyingInstr()->getOpcode(); }
+ /// Return the scalar result type of the replicated instruction. Note that
+ /// for stores this is the void type.
+ Type *getScalarType() const { return getUnderlyingInstr()->getType(); }
+
protected:
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
/// Print the recipe.
@@ -3789,6 +3802,14 @@ class LLVM_ABI_FOR_TEST VPWidenMemoryRecipe : public VPIRMetadata {
/// Return the cost of this VPWidenMemoryRecipe.
InstructionCost computeCost(ElementCount VF, VPCostContext &Ctx) const;
+ /// Return the cost of a masked widened memory operation with opcode \p Opcode
+ /// (Load or Store), vectorized data type \p DataTy, alignment \p Alignment and
+ /// address space \p AS. This is a static helper so callers can price a masked
+ /// memory access without first constructing a recipe.
+ static InstructionCost computeMaskedCost(unsigned Opcode, Type *DataTy,
+ Align Alignment, unsigned AS,
+ VPCostContext &Ctx);
+
Instruction &getIngredient() const { return Ingredient; }
};
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 146dbc3121e54..68bee7dc7439a 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -655,7 +655,6 @@ unsigned VPInstruction::getNumOperandsForOpcode() const {
case Instruction::Store:
case VPInstruction::BranchOnCount:
case VPInstruction::BranchOnTwoConds:
- case VPInstruction::BroadcastLane:
case VPInstruction::FirstOrderRecurrenceSplice:
case VPInstruction::LogicalAnd:
case VPInstruction::LogicalOr:
@@ -732,38 +731,6 @@ static Instruction::BinaryOps getSubRecurOpcode(RecurKind Kind) {
llvm_unreachable("RecurKind should be Sub/FSub.");
}
-/// Broadcast lane \p LaneNo of \p Vec across the result vector.
-/// - Fixed : shufflevector with mask <i32 N, ...>.
-/// - Scalable, lane 0 : shufflevector with zeroinitializer mask.
-/// - Scalable, lane k>0 : extractelement + CreateVectorSplat
-/// (non-zero scalable shuffle masks are rejected
-/// by the verifier today; see
-/// ShuffleVectorInst::isValidOperands).
-static Value *generateBroadcastLane(IRBuilderBase &Builder, Value *Vec,
- unsigned LaneNo) {
- auto *VecTy = cast<VectorType>(Vec->getType());
- Value *Poison = PoisonValue::get(Vec->getType());
-
- if (isa<ScalableVectorType>(VecTy)) {
- if (LaneNo == 0) {
- Type *I32Ty = Type::getInt32Ty(VecTy->getContext());
- auto *MaskTy = VectorType::get(I32Ty, VecTy->getElementCount());
- return Builder.CreateShuffleVector(Vec, Poison,
- Constant::getNullValue(MaskTy),
- "broadcast.lane");
- }
- Value *Scalar =
- Builder.CreateExtractElement(Vec, Builder.getInt32(LaneNo));
- return Builder.CreateVectorSplat(VecTy->getElementCount(), Scalar,
- "broadcast.lane");
- }
-
- auto *FVT = cast<FixedVectorType>(VecTy);
- SmallVector<int, 16> ShufMask(FVT->getNumElements(),
- static_cast<int>(LaneNo));
- return Builder.CreateShuffleVector(Vec, Poison, ShufMask, "broadcast.lane");
-}
-
Value *VPInstruction::generate(VPTransformState &State) {
IRBuilderBase &Builder = State.Builder;
@@ -1125,12 +1092,6 @@ Value *VPInstruction::generate(VPTransformState &State) {
return State.get(getOperand(0), true);
case VPInstruction::Reverse:
return Builder.CreateVectorReverse(State.get(getOperand(0)), "reverse");
- case VPInstruction::BroadcastLane: {
- Value *Idx = State.get(getOperand(0), /*IsScalar*/ true);
- Value *Vec = State.get(getOperand(1));
- unsigned LaneNo = cast<ConstantInt>(Idx)->getZExtValue();
- return generateBroadcastLane(Builder, Vec, LaneNo);
- }
case VPInstruction::ExtractLastActive: {
Value *Result = State.get(getOperand(0), /*IsScalar=*/true);
for (unsigned Idx = 1; Idx < getNumOperands(); Idx += 2) {
@@ -1356,6 +1317,18 @@ InstructionCost VPRecipeWithIRFlags::getCostForRecipeWithOpcode(
llvm_unreachable("called for unsupported opcode");
}
+InstructionCost VPInstruction::computeAnyOfCost(VectorType *MaskTy,
+ VPCostContext &Ctx) {
+ return Ctx.TTI.getArithmeticReductionCost(Instruction::Or, MaskTy,
+ std::nullopt, Ctx.CostKind);
+}
+
+InstructionCost VPInstruction::computeExtractElementCost(VectorType *VecTy,
+ VPCostContext &Ctx) {
+ return Ctx.TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy,
+ Ctx.CostKind);
+}
+
InstructionCost VPInstruction::computeCost(ElementCount VF,
VPCostContext &Ctx) const {
if (Instruction::isBinaryOp(getOpcode())) {
@@ -1395,14 +1368,13 @@ InstructionCost VPInstruction::computeCost(ElementCount VF,
}
// Add on the cost of extracting the element.
- auto *VecTy = toVectorTy(getOperand(0)->getScalarType(), VF);
- return Ctx.TTI.getVectorInstrCost(Instruction::ExtractElement, VecTy,
- Ctx.CostKind);
+ auto *VecTy =
+ cast<VectorType>(toVectorTy(getOperand(0)->getScalarType(), VF));
+ return computeExtractElementCost(VecTy, Ctx);
}
case VPInstruction::AnyOf: {
- auto *VecTy = toVectorTy(this->getScalarType(), VF);
- return Ctx.TTI.getArithmeticReductionCost(
- Instruction::Or, cast<VectorType>(VecTy), std::nullopt, Ctx.CostKind);
+ auto *VecTy = cast<VectorType>(toVectorTy(this->getScalarType(), VF));
+ return computeAnyOfCost(VecTy, Ctx);
}
case VPInstruction::FirstActiveLane: {
Type *Ty = this->getScalarType();
@@ -1484,19 +1456,6 @@ InstructionCost VPInstruction::computeCost(ElementCount VF,
VectorTy, /*Mask=*/{}, Ctx.CostKind,
/*Index=*/0);
}
- case VPInstruction::BroadcastLane: {
- assert(VF.isVector() && "BroadcastLane requires a vector VF");
- Type *EltTy = getOperand(1)->getScalarType();
- auto *VectorTy = cast<VectorType>(toVectorTy(EltTy, VF));
- // Forward the constant lane index so targets can specialise lane-0.
- unsigned LaneIdx = 0;
- if (auto *LiveInIdx = getOperand(0)->getLiveInIRValue())
- if (auto *CI = dyn_cast<ConstantInt>(LiveInIdx))
- LaneIdx = CI->getZExtValue();
- return Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VectorTy,
- VectorTy, /*Mask=*/{}, Ctx.CostKind,
- /*Index=*/LaneIdx);
- }
case VPInstruction::ExtractLastLane: {
// Add on the cost of extracting the element.
auto *VecTy = toVectorTy(getOperand(0)->getScalarType(), VF);
@@ -1759,7 +1718,6 @@ bool VPInstruction::usesFirstLaneOnly(const VPValue *Op) const {
return false;
case VPInstruction::ExitingIVValue:
case VPInstruction::ExtractLane:
- case VPInstruction::BroadcastLane:
return Op == getOperand(0);
};
llvm_unreachable("switch should return");
@@ -1892,9 +1850,6 @@ void VPInstruction::printRecipe(raw_ostream &O, const Twine &Indent,
case VPInstruction::Reverse:
O << "reverse";
break;
- case VPInstruction::BroadcastLane:
- O << "broadcast-lane";
- break;
case VPInstruction::Unpack:
O << "unpack";
break;
@@ -4161,6 +4116,17 @@ const VPRecipeBase *VPWidenStoreRecipe::getAsRecipe() const { return this; }
VPRecipeBase *VPWidenStoreEVLRecipe::getAsRecipe() { return this; }
const VPRecipeBase *VPWidenStoreEVLRecipe::getAsRecipe() const { return this; }
+InstructionCost VPWidenMemoryRecipe::computeMaskedCost(unsigned Opcode,
+ Type *DataTy,
+ Align Alignment,
+ unsigned AS,
+ VPCostContext &Ctx) {
+ unsigned IID = Opcode == Instruction::Load ? Intrinsic::masked_load
+ : Intrinsic::masked_store;
+ return Ctx.TTI.getMemIntrinsicInstrCost(
+ MemIntrinsicCostAttributes(IID, DataTy, Alignment, AS), Ctx.CostKind);
+}
+
InstructionCost VPWidenMemoryRecipe::computeCost(ElementCount VF,
VPCostContext &Ctx) const {
const VPRecipeBase *R = getAsRecipe();
@@ -4198,10 +4164,7 @@ InstructionCost VPWidenMemoryRecipe::computeCost(ElementCount VF,
InstructionCost Cost = 0;
if (IsMasked) {
- unsigned IID = isa<VPWidenLoadRecipe>(R) ? Intrinsic::masked_load
- : Intrinsic::masked_store;
- Cost += Ctx.TTI.getMemIntrinsicInstrCost(
- MemIntrinsicCostAttributes(IID, Ty, Alignment, AS), Ctx.CostKind);
+ Cost += computeMaskedCost(Opcode, Ty, Alignment, AS, Ctx);
} else {
TTI::OperandValueInfo OpInfo = Ctx.getOperandInfo(
isa<VPWidenLoadRecipe, VPWidenLoadEVLRecipe>(R) ? R->getOperand(0)
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index 0f58276040715..3ed786d3343f5 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -2304,42 +2304,6 @@ struct VPCSEDenseMapInfo : public DenseMapInfo<VPSingleDefRecipe *> {
};
} // end anonymous namespace
-/// CSE duplicate masked VPWidenLoadRecipes (which are not VPSingleDefRecipes
-/// and so are missed by the generic cse()). Reset on any write within the
-/// block; unmasked loads are left to downstream EarlyCSE/GVN.
-void VPlanTransforms::cseUniformMemoryReads(VPlan &Plan) {
- ReversePostOrderTraversal<VPBlockDeepTraversalWrapper<VPBlockBase *>> RPOT(
- Plan.getEntry());
- for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(RPOT)) {
- SmallVector<VPWidenLoadRecipe *, 4> Candidates;
- for (VPRecipeBase &R : make_early_inc_range(*VPBB)) {
- if (R.mayWriteToMemory()) {
- Candidates.clear();
- continue;
- }
- auto *Load = dyn_cast<VPWidenLoadRecipe>(&R);
- if (!Load || !Load->isMasked())
- continue;
- VPWidenLoadRecipe *Match = nullptr;
- for (VPWidenLoadRecipe *C : Candidates) {
- if (!equal(C->operands(), Load->operands()))
- continue;
- // Same ingredient => same alignment/metadata/type.
- if (&C->getIngredient() != &Load->getIngredient())
- continue;
- Match = C;
- break;
- }
- if (Match) {
- Load->getVPSingleValue()->replaceAllUsesWith(Match->getVPSingleValue());
- Load->eraseFromParent();
- continue;
- }
- Candidates.push_back(Load);
- }
- }
-}
-
/// Perform a common-subexpression-elimination of VPSingleDefRecipes on the \p
/// Plan.
void VPlanTransforms::cse(VPlan &Plan) {
@@ -3888,8 +3852,8 @@ static cl::opt<bool> EnableMaskedInvariantLoad(
cl::desc("Rewrite predicated loads from loop-invariant addresses as a "
"single-lane masked load + broadcast"));
-void VPlanTransforms::widenPredicatedInvariantLoads(
- VPlan &Plan, const TargetTransformInfo &TTI) {
+void VPlanTransforms::widenPredicatedInvariantLoads(VPlan &Plan,
+ VPCostContext &CostCtx) {
if (!EnableMaskedInvariantLoad)
return;
@@ -3911,9 +3875,12 @@ void VPlanTransforms::widenPredicatedInvariantLoads(
VPValue *Addr = RepR->getOperand(0);
if (!Addr->isDefinedOutsideLoopRegions())
continue;
+ // Prefer information from the recipe; alignment/address space are only
+ // available on the underlying load.
auto *LI = cast<LoadInst>(RepR->getUnderlyingInstr());
- if (!TTI.isLegalMaskedLoad(LI->getType(), LI->getAlign(),
- LI->getPointerAddressSpace()))
+ if (!ForceTargetSupportsMaskedMemoryOps &&
+ !CostCtx.TTI.isLegalMaskedLoad(RepR->getScalarType(), LI->getAlign(),
+ LI->getPointerAddressSpace()))
continue;
Worklist.push_back(RepR);
}
@@ -3922,7 +3889,34 @@ void VPlanTransforms::widenPredicatedInvariantLoads(
for (VPReplicateRecipe *RepR : Worklist) {
auto *LI = cast<LoadInst>(RepR->getUnderlyingInstr());
LLVMContext &Ctx = LI->getContext();
- DebugLoc DL = LI->getDebugLoc();
+ Type *LoadTy = RepR->getScalarType();
+ Align Alignment = LI->getAlign();
+ unsigned AS = LI->getPointerAddressSpace();
+ Type *I1Ty = Type::getInt1Ty(Ctx);
+
+ // Only rewrite when the masked-load + broadcast lowering is no more
+ // expensive than the scalarized predicated load for every vector VF.
+ bool Beneficial = true;
+ for (ElementCount VF : Plan.vectorFactors()) {
+ if (!VF.isVector())
+ continue;
+ auto *MaskTy = cast<VectorType>(toVectorTy(I1Ty, VF));
+ auto *VecTy = cast<VectorType>(toVectorTy(LoadTy, VF));
+ InstructionCost NewCost =
+ VPInstruction::computeAnyOfCost(MaskTy, CostCtx) +
+ VPWidenLoadRecipe::computeMaskedCost(Instruction::Load, VecTy,
+ Alignment, AS, CostCtx) +
+ VPInstruction::computeExtractElementCost(VecTy, CostCtx);
+ InstructionCost OldCost = RepR->computeCost(VF, CostCtx);
+ if (NewCost > OldCost) {
+ Beneficial = false;
+ break;
+ }
+ }
+ if (!Beneficial)
+ continue;
+
+ DebugLoc DL = RepR->getDebugLoc();
VPValue *Addr = RepR->getOperand(0);
VPValue *Mask = RepR->getMask();
@@ -3930,11 +3924,10 @@ void VPlanTransforms::widenPredicatedInvariantLoads(
// any = AnyOf(mask). Unroll appends per-part masks, so this becomes a
// single OR-reduction across all UF parts.
- VPValue *AnyActive =
- Builder.createNaryOp(VPInstruction::AnyOf, {Mask}, DL);
+ VPValue *AnyActive = Builder.createNaryOp(VPInstruction::AnyOf, {Mask}, DL);
- // vmsk = insertelement <VF x i1> zeroinitializer, any, 0
- Type *I1Ty = Type::getInt1Ty(Ctx);
+ // Build a mask with `any` in lane 0 and false elsewhere:
+ // vmsk = insertelement <VF x i1> zeroinitializer, any, 0
VPValue *FalseI1 = Plan.getOrAddLiveIn(ConstantInt::getFalse(I1Ty));
VPValue *ZeroVMask =
Builder.createNaryOp(VPInstruction::Broadcast, {FalseI1}, DL);
@@ -3944,17 +3937,18 @@ void VPlanTransforms::widenPredicatedInvariantLoads(
Instruction::InsertElement, {ZeroVMask, AnyActive, Lane0Idx}, DL);
// Consecutive=true selects the masked-load (not masked-gather) lowering in
- // VPWidenLoadRecipe::execute. The access is stride-0, not stride-1; the
- // BroadcastLane below splats lane 0 across all VF lanes.
+ // VPWidenLoadRecipe::execute. The access is stride-0, not stride-1; lane 0
+ // holds the loaded value and is broadcast across all VF lanes below.
auto *MaskedLoad = new VPWidenLoadRecipe(*LI, Addr, VMask,
/*Consecutive*/ true, *RepR, DL);
Builder.insert(MaskedLoad);
- auto *Bcast = new VPInstruction(
- VPInstruction::BroadcastLane,
- {Lane0Idx, MaskedLoad->getVPSingleValue()}, /*Flags*/ {},
- /*MD*/ {}, DL);
- Builder.insert(Bcast);
+ // Broadcast lane 0 of the masked load across all lanes, reusing the
+ // existing extractelement + broadcast recipes.
+ VPValue *Lane0 = Builder.createNaryOp(
+ Instruction::ExtractElement, {MaskedLoad->getVPSingleValue(), Lane0Idx},
+ DL);
+ VPValue *Bcast = Builder.createNaryOp(VPInstruction::Broadcast, {Lane0}, DL);
RepR->replaceAllUsesWith(Bcast);
RepR->eraseFromParent();
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index ba1c6e7428a9b..d2f409f535660 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -40,6 +40,7 @@ struct VFRange;
LLVM_ABI_FOR_TEST extern cl::opt<bool> VerifyEachVPlan;
LLVM_ABI_FOR_TEST extern cl::opt<bool> EnableWideActiveLaneMask;
+LLVM_ABI_FOR_TEST extern cl::opt<bool> ForceTargetSupportsMaskedMemoryOps;
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
LLVM_ABI_FOR_TEST extern cl::opt<bool> VPlanPrintBeforeAll;
@@ -413,10 +414,6 @@ struct VPlanTransforms {
/// Perform common-subexpression-elimination on \p Plan.
static void cse(VPlan &Plan);
- /// CSE masked VPWidenLoadRecipes with identical operands inside a basic
- /// block, ignoring candidates across an intervening write. No AA needed.
- static void cseUniformMemoryReads(VPlan &Plan);
-
/// If there's a single exit block, optimize its phi recipes that use exiting
/// IV values by feeding them precomputed end values instead, possibly taken
/// one step backwards.
@@ -435,10 +432,11 @@ struct VPlanTransforms {
/// Rewrite predicated VPReplicateRecipe loads from a loop-invariant address
/// into a single-lane masked load + broadcast pattern. Only applied when
- /// the target reports isLegalMaskedLoad for the element type. Gated by
- /// -enable-masked-invariant-load.
+ /// the target reports isLegalMaskedLoad for the element type and when the new
+ /// lowering is not more expensive than the scalarized predicated load for all
+ /// vector VFs of \p Plan. Gated by -enable-masked-invariant-load.
static void widenPredicatedInvariantLoads(VPlan &Plan,
- const TargetTransformInfo &TTI);
+ VPCostContext &CostCtx);
/// Sink predicated stores to the same address with complementary predicates
/// (P and NOT P) to an unconditional store with select recipes for the
diff --git a/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp b/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp
index b50faf9b0b5ea..a4d6cc0ecb480 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp
@@ -40,10 +40,6 @@ class VPlanVerifier {
/// Verify that \p LastActiveLane's operand is guaranteed to be a prefix-mask.
bool verifyLastActiveLaneRecipe(const VPInstruction &LastActiveLane) const;
- /// Operand 0 of \p BroadcastLane must be a live-in constant integer
- /// (shufflevector masks are constant-only).
- bool verifyBroadcastLaneRecipe(const VPInstruction &BroadcastLane) const;
-
bool verifyVPBasicBlock(const VPBasicBlock *VPBB);
bool verifyBlock(const VPBlockBase *VPB);
@@ -228,28 +224,6 @@ bool VPlanVerifier::verifyLastActiveLaneRecipe(
return true;
}
-bool VPlanVerifier::verifyBroadcastLaneRecipe(
- const VPInstruction &BroadcastLane) const {
- assert(BroadcastLane.getOpcode() == VPInstruction::BroadcastLane &&
- "must be called with VPInstruction::BroadcastLane");
-
- if (BroadcastLane.getNumOperands() != 2) {
- errs() << "BroadcastLane must have exactly two operands "
- "(lane index, vector)\n";
- return false;
- }
-
- const VPValue *LaneIdx = BroadcastLane.getOperand(0);
- Value *LiveInIdx = LaneIdx->getLiveInIRValue();
- if (!LiveInIdx || !isa<ConstantInt>(LiveInIdx)) {
- errs() << "BroadcastLane operand 0 must be a constant integer lane "
- "index (live-in ConstantInt)\n";
- return false;
- }
-
- return true;
-}
-
bool VPlanVerifier::verifyVPBasicBlock(const VPBasicBlock *VPBB) {
if (!verifyPhiRecipes(VPBB))
return false;
@@ -356,10 +330,6 @@ bool VPlanVerifier::verifyVPBasicBlock(const VPBasicBlock *VPBB) {
if (!verifyLastActiveLaneRecipe(*VPI))
return false;
break;
- case VPInstruction::BroadcastLane:
- if (!verifyBroadcastLaneRecipe(*VPI))
- return false;
- break;
default:
break;
}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll
deleted file mode 100644
index 43be17e82ec54..0000000000000
--- a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-multiple.ll
+++ /dev/null
@@ -1,93 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=loop-vectorize -enable-masked-invariant-load -force-vector-width=4 \
-; RUN: -force-vector-interleave=1 -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S \
-; RUN: | FileCheck %s
-
-;; Two distinct loop-invariant pointers loaded under the same predicate in the
-;; same block. Each load is independently lowered to a single-lane masked load
-;; + broadcast, but they share the SAME AnyOf(mask) and lane-0 mask material
-;; thanks to the standard VPlan CSE that runs before unrolling.
-;;
-;; for (i = 0; i < n; ++i)
-;; if (cond[i])
-;; a[i] = *inv1 + *inv2;
-
-define void @cond_invar_load_two_ptrs(ptr noalias %a, ptr noalias %cond, ptr noalias %inv1, ptr noalias %inv2, i64 %n) {
-; CHECK-LABEL: @cond_invar_load_two_ptrs(
-; CHECK-NEXT: entry:
-; CHECK-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N:%.*]], 4
-; CHECK-NEXT: br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
-; CHECK: vector.ph:
-; CHECK-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
-; CHECK-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
-; CHECK-NEXT: br label [[VECTOR_BODY:%.*]]
-; CHECK: vector.body:
-; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
-; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
-; CHECK-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = freeze <4 x i1> [[TMP1]]
-; CHECK-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
-; CHECK-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
-; CHECK-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV1:%.*]], <4 x i1> [[TMP4]], <4 x i32> poison)
-; CHECK-NEXT: [[BROADCAST_LANE:%.*]] = shufflevector <4 x i32> [[WIDE_MASKED_LOAD]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[WIDE_MASKED_LOAD1:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV2:%.*]], <4 x i1> [[TMP4]], <4 x i32> poison)
-; CHECK-NEXT: [[BROADCAST_LANE2:%.*]] = shufflevector <4 x i32> [[WIDE_MASKED_LOAD1]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP5:%.*]] = add <4 x i32> [[BROADCAST_LANE]], [[BROADCAST_LANE2]]
-; CHECK-NEXT: [[TMP6:%.*]] = getelementptr i32, ptr [[A:%.*]], i64 [[INDEX]]
-; CHECK-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[TMP5]], ptr align 4 [[TMP6]], <4 x i1> [[TMP1]])
-; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-NEXT: br i1 [[TMP7]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
-; CHECK: middle.block:
-; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
-; CHECK-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
-; CHECK: scalar.ph:
-; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
-; CHECK-NEXT: br label [[FOR_BODY:%.*]]
-; CHECK: for.body:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[I_NEXT:%.*]], [[FOR_INC:%.*]] ]
-; CHECK-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
-; CHECK-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
-; CHECK-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
-; CHECK-NEXT: br i1 [[TOBOOL]], label [[IF_THEN:%.*]], label [[FOR_INC]]
-; CHECK: if.then:
-; CHECK-NEXT: [[V1:%.*]] = load i32, ptr [[INV1]], align 4
-; CHECK-NEXT: [[V2:%.*]] = load i32, ptr [[INV2]], align 4
-; CHECK-NEXT: [[SUM:%.*]] = add i32 [[V1]], [[V2]]
-; CHECK-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: store i32 [[SUM]], ptr [[A_GEP]], align 4
-; CHECK-NEXT: br label [[FOR_INC]]
-; CHECK: for.inc:
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
-; CHECK-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
-; CHECK: for.end:
-; CHECK-NEXT: ret void
-;
-entry:
- br label %for.body
-
-for.body:
- %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
- %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
- %cv = load i32, ptr %cond.gep, align 4
- %tobool = icmp ne i32 %cv, 0
- br i1 %tobool, label %if.then, label %for.inc
-
-if.then:
- %v1 = load i32, ptr %inv1, align 4
- %v2 = load i32, ptr %inv2, align 4
- %sum = add i32 %v1, %v2
- %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
- store i32 %sum, ptr %a.gep, align 4
- br label %for.inc
-
-for.inc:
- %i.next = add nuw nsw i64 %i, 1
- %exitcond = icmp eq i64 %i.next, %n
- br i1 %exitcond, label %for.end, label %for.body
-
-for.end:
- ret void
-}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll
deleted file mode 100644
index e502761b8a7e2..0000000000000
--- a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load-not-invariant.ll
+++ /dev/null
@@ -1,64 +0,0 @@
-; RUN: opt < %s -passes=loop-vectorize -enable-masked-invariant-load -force-vector-width=4 \
-; RUN: -force-vector-interleave=1 -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S \
-; RUN: | FileCheck %s
-
-;; Negative test for CM_CondInvar.
-;;
-;; The conditional load's pointer (%src + i) is loop-VARIANT, so even with
-;; -enable-masked-invariant-load enabled, getConditionalInvarCost() returns
-;; Invalid and the cost model picks the regular consecutive masked load
-;; (CM_Widen).
-;;
-;; for (i = 0; i < n; ++i)
-;; if (cond[i])
-;; a[i] = src[i];
-;;
-;; The output:
-;; * Must contain a regular widened consecutive masked load on src + index.
-;; * Must NOT contain any AnyOf-style i1 OR-reduction.
-;; * Must NOT contain a broadcast.lane shuffle (the CondInvar lowering
-;; marker).
-;; * Must NOT contain a masked.load whose pointer is the loop-invariant
-;; %src argument.
-
-define void @cond_load_not_invariant(ptr noalias %a, ptr noalias %src, ptr noalias %cond, i64 %n) {
-; CHECK-LABEL: define void @cond_load_not_invariant(
-
-; The vector body widens the consecutive masked load on src[index].
-; CHECK: vector.body:
-; CHECK: [[SRC_GEP_V:%.*]] = getelementptr i32, ptr %src, i64 {{%.*}}
-; CHECK: call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[SRC_GEP_V]],
-
-; The CondInvar lowering must NOT be applied.
-; CHECK-NOT: call i1 @llvm.vector.reduce.or
-; CHECK-NOT: %broadcast.lane = shufflevector
-; CHECK-NOT: call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 %src,
-
-; CHECK: middle.block:
-; CHECK: ret void
-
-entry:
- br label %for.body
-
-for.body:
- %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
- %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
- %cv = load i32, ptr %cond.gep, align 4
- %tobool = icmp ne i32 %cv, 0
- br i1 %tobool, label %if.then, label %for.inc
-
-if.then:
- %src.gep = getelementptr inbounds i32, ptr %src, i64 %i
- %v = load i32, ptr %src.gep, align 4
- %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
- store i32 %v, ptr %a.gep, align 4
- br label %for.inc
-
-for.inc:
- %i.next = add nuw nsw i64 %i, 1
- %exitcond = icmp eq i64 %i.next, %n
- br i1 %exitcond, label %for.end, label %for.body
-
-for.end:
- ret void
-}
diff --git a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll b/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll
deleted file mode 100644
index 4a66d907675d0..0000000000000
--- a/llvm/test/Transforms/LoopVectorize/X86/enable-masked-invariant-load.ll
+++ /dev/null
@@ -1,169 +0,0 @@
-; NOTE: Assertions have been autogenerated by utils/update_test_checks.py
-; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
-; RUN: -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S | FileCheck %s --check-prefix=CHECK-OFF
-; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
-; RUN: -enable-masked-invariant-load -mtriple=x86_64-unknown-linux-gnu -mattr=avx2 -S | FileCheck %s --check-prefix=CHECK-ON
-
-;; Conditional load from a loop-invariant pointer.
-;;
-;; for (i = 0; i < n; ++i)
-;; if (cond[i])
-;; a[i] = *inv;
-;;
-;; The pointer %inv is loop-invariant and the load is predicated by %tobool.
-;; Without -enable-masked-invariant-load the cost model falls back to per-lane
-;; scalarised predicated loads (CM_Scalarize) - one pred.load.if/continue
-;; chain per VF lane.
-;;
-;; With -enable-masked-invariant-load the load becomes:
-;; * AnyOf reduction across the block mask,
-;; * lane-0 mask materialisation (zeroinitializer + insertelement i1),
-;; * one masked.load from %inv,
-;; * shufflevector broadcast of lane 0 to all VF lanes.
-
-define void @cond_invar_load(ptr noalias %a, ptr noalias %cond, ptr noalias %inv, i64 %n) {
-; CHECK-OFF-LABEL: @cond_invar_load(
-; CHECK-OFF-NEXT: entry:
-; CHECK-OFF-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N:%.*]], 4
-; CHECK-OFF-NEXT: br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
-; CHECK-OFF: vector.ph:
-; CHECK-OFF-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
-; CHECK-OFF-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
-; CHECK-OFF-NEXT: br label [[VECTOR_BODY:%.*]]
-; CHECK-OFF: vector.body:
-; CHECK-OFF-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[PRED_LOAD_CONTINUE6:%.*]] ]
-; CHECK-OFF-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
-; CHECK-OFF-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
-; CHECK-OFF-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
-; CHECK-OFF-NEXT: [[TMP2:%.*]] = extractelement <4 x i1> [[TMP1]], i64 0
-; CHECK-OFF-NEXT: br i1 [[TMP2]], label [[PRED_LOAD_IF:%.*]], label [[PRED_LOAD_CONTINUE:%.*]]
-; CHECK-OFF: pred.load.if:
-; CHECK-OFF-NEXT: [[TMP3:%.*]] = load i32, ptr [[INV:%.*]], align 4
-; CHECK-OFF-NEXT: [[TMP4:%.*]] = insertelement <4 x i32> poison, i32 [[TMP3]], i64 0
-; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE]]
-; CHECK-OFF: pred.load.continue:
-; CHECK-OFF-NEXT: [[TMP5:%.*]] = phi <4 x i32> [ poison, [[VECTOR_BODY]] ], [ [[TMP4]], [[PRED_LOAD_IF]] ]
-; CHECK-OFF-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP1]], i64 1
-; CHECK-OFF-NEXT: br i1 [[TMP6]], label [[PRED_LOAD_IF1:%.*]], label [[PRED_LOAD_CONTINUE2:%.*]]
-; CHECK-OFF: pred.load.if1:
-; CHECK-OFF-NEXT: [[TMP7:%.*]] = load i32, ptr [[INV]], align 4
-; CHECK-OFF-NEXT: [[TMP8:%.*]] = insertelement <4 x i32> [[TMP5]], i32 [[TMP7]], i64 1
-; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE2]]
-; CHECK-OFF: pred.load.continue2:
-; CHECK-OFF-NEXT: [[TMP9:%.*]] = phi <4 x i32> [ [[TMP5]], [[PRED_LOAD_CONTINUE]] ], [ [[TMP8]], [[PRED_LOAD_IF1]] ]
-; CHECK-OFF-NEXT: [[TMP10:%.*]] = extractelement <4 x i1> [[TMP1]], i64 2
-; CHECK-OFF-NEXT: br i1 [[TMP10]], label [[PRED_LOAD_IF3:%.*]], label [[PRED_LOAD_CONTINUE4:%.*]]
-; CHECK-OFF: pred.load.if3:
-; CHECK-OFF-NEXT: [[TMP11:%.*]] = load i32, ptr [[INV]], align 4
-; CHECK-OFF-NEXT: [[TMP12:%.*]] = insertelement <4 x i32> [[TMP9]], i32 [[TMP11]], i64 2
-; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE4]]
-; CHECK-OFF: pred.load.continue4:
-; CHECK-OFF-NEXT: [[TMP13:%.*]] = phi <4 x i32> [ [[TMP9]], [[PRED_LOAD_CONTINUE2]] ], [ [[TMP12]], [[PRED_LOAD_IF3]] ]
-; CHECK-OFF-NEXT: [[TMP14:%.*]] = extractelement <4 x i1> [[TMP1]], i64 3
-; CHECK-OFF-NEXT: br i1 [[TMP14]], label [[PRED_LOAD_IF5:%.*]], label [[PRED_LOAD_CONTINUE6]]
-; CHECK-OFF: pred.load.if5:
-; CHECK-OFF-NEXT: [[TMP15:%.*]] = load i32, ptr [[INV]], align 4
-; CHECK-OFF-NEXT: [[TMP16:%.*]] = insertelement <4 x i32> [[TMP13]], i32 [[TMP15]], i64 3
-; CHECK-OFF-NEXT: br label [[PRED_LOAD_CONTINUE6]]
-; CHECK-OFF: pred.load.continue6:
-; CHECK-OFF-NEXT: [[TMP17:%.*]] = phi <4 x i32> [ [[TMP13]], [[PRED_LOAD_CONTINUE4]] ], [ [[TMP16]], [[PRED_LOAD_IF5]] ]
-; CHECK-OFF-NEXT: [[TMP18:%.*]] = getelementptr i32, ptr [[A:%.*]], i64 [[INDEX]]
-; CHECK-OFF-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[TMP17]], ptr align 4 [[TMP18]], <4 x i1> [[TMP1]])
-; CHECK-OFF-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-OFF-NEXT: [[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-OFF-NEXT: br i1 [[TMP19]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
-; CHECK-OFF: middle.block:
-; CHECK-OFF-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
-; CHECK-OFF-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
-; CHECK-OFF: scalar.ph:
-; CHECK-OFF-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
-; CHECK-OFF-NEXT: br label [[FOR_BODY:%.*]]
-; CHECK-OFF: for.body:
-; CHECK-OFF-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[I_NEXT:%.*]], [[FOR_INC:%.*]] ]
-; CHECK-OFF-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
-; CHECK-OFF-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
-; CHECK-OFF-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
-; CHECK-OFF-NEXT: br i1 [[TOBOOL]], label [[IF_THEN:%.*]], label [[FOR_INC]]
-; CHECK-OFF: if.then:
-; CHECK-OFF-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
-; CHECK-OFF-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-OFF-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
-; CHECK-OFF-NEXT: br label [[FOR_INC]]
-; CHECK-OFF: for.inc:
-; CHECK-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
-; CHECK-OFF-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
-; CHECK-OFF-NEXT: br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
-; CHECK-OFF: for.end:
-; CHECK-OFF-NEXT: ret void
-;
-; CHECK-ON-LABEL: @cond_invar_load(
-; CHECK-ON-NEXT: entry:
-; CHECK-ON-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N:%.*]], 4
-; CHECK-ON-NEXT: br i1 [[MIN_ITERS_CHECK]], label [[SCALAR_PH:%.*]], label [[VECTOR_PH:%.*]]
-; CHECK-ON: vector.ph:
-; CHECK-ON-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
-; CHECK-ON-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
-; CHECK-ON-NEXT: br label [[VECTOR_BODY:%.*]]
-; CHECK-ON: vector.body:
-; CHECK-ON-NEXT: [[INDEX:%.*]] = phi i64 [ 0, [[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], [[VECTOR_BODY]] ]
-; CHECK-ON-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND:%.*]], i64 [[INDEX]]
-; CHECK-ON-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
-; CHECK-ON-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
-; CHECK-ON-NEXT: [[TMP2:%.*]] = freeze <4 x i1> [[TMP1]]
-; CHECK-ON-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
-; CHECK-ON-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
-; CHECK-ON-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV:%.*]], <4 x i1> [[TMP4]], <4 x i32> poison)
-; CHECK-ON-NEXT: [[BROADCAST_LANE:%.*]] = shufflevector <4 x i32> [[WIDE_MASKED_LOAD]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-ON-NEXT: [[TMP5:%.*]] = getelementptr i32, ptr [[A:%.*]], i64 [[INDEX]]
-; CHECK-ON-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[BROADCAST_LANE]], ptr align 4 [[TMP5]], <4 x i1> [[TMP1]])
-; CHECK-ON-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
-; CHECK-ON-NEXT: [[TMP6:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
-; CHECK-ON-NEXT: br i1 [[TMP6]], label [[MIDDLE_BLOCK:%.*]], label [[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
-; CHECK-ON: middle.block:
-; CHECK-ON-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
-; CHECK-ON-NEXT: br i1 [[CMP_N]], label [[FOR_END:%.*]], label [[SCALAR_PH]]
-; CHECK-ON: scalar.ph:
-; CHECK-ON-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], [[MIDDLE_BLOCK]] ], [ 0, [[ENTRY:%.*]] ]
-; CHECK-ON-NEXT: br label [[FOR_BODY:%.*]]
-; CHECK-ON: for.body:
-; CHECK-ON-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], [[SCALAR_PH]] ], [ [[I_NEXT:%.*]], [[FOR_INC:%.*]] ]
-; CHECK-ON-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
-; CHECK-ON-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
-; CHECK-ON-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
-; CHECK-ON-NEXT: br i1 [[TOBOOL]], label [[IF_THEN:%.*]], label [[FOR_INC]]
-; CHECK-ON: if.then:
-; CHECK-ON-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
-; CHECK-ON-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-ON-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
-; CHECK-ON-NEXT: br label [[FOR_INC]]
-; CHECK-ON: for.inc:
-; CHECK-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
-; CHECK-ON-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
-; CHECK-ON-NEXT: br i1 [[EXITCOND]], label [[FOR_END]], label [[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
-; CHECK-ON: for.end:
-; CHECK-ON-NEXT: ret void
-;
-entry:
- br label %for.body
-
-for.body:
- %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
- %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
- %cv = load i32, ptr %cond.gep, align 4
- %tobool = icmp ne i32 %cv, 0
- br i1 %tobool, label %if.then, label %for.inc
-
-if.then:
- %v = load i32, ptr %inv, align 4
- %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
- store i32 %v, ptr %a.gep, align 4
- br label %for.inc
-
-for.inc:
- %i.next = add nuw nsw i64 %i, 1
- %exitcond = icmp eq i64 %i.next, %n
- br i1 %exitcond, label %for.end, label %for.body
-
-for.end:
- ret void
-}
diff --git a/llvm/test/Transforms/LoopVectorize/masked-invariant-load-gate.ll b/llvm/test/Transforms/LoopVectorize/masked-invariant-load-gate.ll
new file mode 100644
index 0000000000000..fbe72d0f4c542
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/masked-invariant-load-gate.ll
@@ -0,0 +1,254 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=2 -force-vector-interleave=1 \
+; RUN: -force-target-supports-masked-memory-ops -enable-masked-invariant-load -S \
+; RUN: | FileCheck %s --check-prefixes=REJECT
+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
+; RUN: -force-target-supports-masked-memory-ops -enable-masked-invariant-load -S \
+; RUN: | FileCheck %s --check-prefixes=ACCEPT
+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
+; RUN: -enable-masked-invariant-load -S \
+; RUN: | FileCheck %s --check-prefixes=NOMASK
+
+;; Gating of widenPredicatedInvariantLoads.
+;;
+;; The masked-load + broadcast lowering is only committed when (a) the target
+;; can lower a masked load of the element type and (b) it is not more expensive
+;; than the scalarized predicated load, for every vector VF the plan covers.
+;; The three runs below isolate each gate on the same loop:
+;;
+;; REJECT (VF=2, masked ops forced legal): the lowering is legal but the cost
+;; gate rejects it -- scalarizing 2 lanes is cheaper than the
+;; any-of-reduction + masked load + lane-0 extract + broadcast -- so the
+;; load stays a per-lane pred.load.if/continue chain.
+;; ACCEPT (VF=4, masked ops forced legal): the scalarized cost grows with the
+;; number of lanes, so the cost gate accepts and the transform fires
+;; (vector.reduce.or + masked.load + broadcast).
+;; NOMASK (VF=4, no forced legality): the default target cannot lower a masked
+;; load, so the legality gate rejects before costing and the load stays
+;; scalarized.
+define void @cond_invar_load_gate(ptr noalias %a, ptr noalias %cond, ptr noalias %inv, i64 %n) {
+; REJECT-LABEL: define void @cond_invar_load_gate(
+; REJECT-SAME: ptr noalias [[A:%.*]], ptr noalias [[COND:%.*]], ptr noalias [[INV:%.*]], i64 [[N:%.*]]) {
+; REJECT-NEXT: [[ENTRY:.*]]:
+; REJECT-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 2
+; REJECT-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; REJECT: [[VECTOR_PH]]:
+; REJECT-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 2
+; REJECT-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; REJECT-NEXT: br label %[[VECTOR_BODY:.*]]
+; REJECT: [[VECTOR_BODY]]:
+; REJECT-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[PRED_LOAD_CONTINUE2:.*]] ]
+; REJECT-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; REJECT-NEXT: [[WIDE_LOAD:%.*]] = load <2 x i32>, ptr [[TMP0]], align 4
+; REJECT-NEXT: [[TMP1:%.*]] = icmp ne <2 x i32> [[WIDE_LOAD]], zeroinitializer
+; REJECT-NEXT: [[TMP2:%.*]] = extractelement <2 x i1> [[TMP1]], i64 0
+; REJECT-NEXT: br i1 [[TMP2]], label %[[PRED_LOAD_IF:.*]], label %[[PRED_LOAD_CONTINUE:.*]]
+; REJECT: [[PRED_LOAD_IF]]:
+; REJECT-NEXT: [[TMP3:%.*]] = load i32, ptr [[INV]], align 4
+; REJECT-NEXT: [[TMP4:%.*]] = insertelement <2 x i32> poison, i32 [[TMP3]], i64 0
+; REJECT-NEXT: br label %[[PRED_LOAD_CONTINUE]]
+; REJECT: [[PRED_LOAD_CONTINUE]]:
+; REJECT-NEXT: [[TMP5:%.*]] = phi <2 x i32> [ poison, %[[VECTOR_BODY]] ], [ [[TMP4]], %[[PRED_LOAD_IF]] ]
+; REJECT-NEXT: [[TMP6:%.*]] = extractelement <2 x i1> [[TMP1]], i64 1
+; REJECT-NEXT: br i1 [[TMP6]], label %[[PRED_LOAD_IF1:.*]], label %[[PRED_LOAD_CONTINUE2]]
+; REJECT: [[PRED_LOAD_IF1]]:
+; REJECT-NEXT: [[TMP7:%.*]] = load i32, ptr [[INV]], align 4
+; REJECT-NEXT: [[TMP8:%.*]] = insertelement <2 x i32> [[TMP5]], i32 [[TMP7]], i64 1
+; REJECT-NEXT: br label %[[PRED_LOAD_CONTINUE2]]
+; REJECT: [[PRED_LOAD_CONTINUE2]]:
+; REJECT-NEXT: [[TMP9:%.*]] = phi <2 x i32> [ [[TMP5]], %[[PRED_LOAD_CONTINUE]] ], [ [[TMP8]], %[[PRED_LOAD_IF1]] ]
+; REJECT-NEXT: [[TMP10:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
+; REJECT-NEXT: call void @llvm.masked.store.v2i32.p0(<2 x i32> [[TMP9]], ptr align 4 [[TMP10]], <2 x i1> [[TMP1]])
+; REJECT-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 2
+; REJECT-NEXT: [[TMP11:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; REJECT-NEXT: br i1 [[TMP11]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; REJECT: [[MIDDLE_BLOCK]]:
+; REJECT-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; REJECT-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; REJECT: [[SCALAR_PH]]:
+; REJECT-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; REJECT-NEXT: br label %[[FOR_BODY:.*]]
+; REJECT: [[FOR_BODY]]:
+; REJECT-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; REJECT-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; REJECT-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; REJECT-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; REJECT-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; REJECT: [[IF_THEN]]:
+; REJECT-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; REJECT-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; REJECT-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; REJECT-NEXT: br label %[[FOR_INC]]
+; REJECT: [[FOR_INC]]:
+; REJECT-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; REJECT-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; REJECT-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; REJECT: [[FOR_END]]:
+; REJECT-NEXT: ret void
+;
+; ACCEPT-LABEL: define void @cond_invar_load_gate(
+; ACCEPT-SAME: ptr noalias [[A:%.*]], ptr noalias [[COND:%.*]], ptr noalias [[INV:%.*]], i64 [[N:%.*]]) {
+; ACCEPT-NEXT: [[ENTRY:.*]]:
+; ACCEPT-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; ACCEPT-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; ACCEPT: [[VECTOR_PH]]:
+; ACCEPT-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; ACCEPT-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; ACCEPT-NEXT: br label %[[VECTOR_BODY:.*]]
+; ACCEPT: [[VECTOR_BODY]]:
+; ACCEPT-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; ACCEPT-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; ACCEPT-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; ACCEPT-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; ACCEPT-NEXT: [[TMP2:%.*]] = freeze <4 x i1> [[TMP1]]
+; ACCEPT-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; ACCEPT-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
+; ACCEPT-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV]], <4 x i1> [[TMP4]], <4 x i32> poison)
+; ACCEPT-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[WIDE_MASKED_LOAD]], i64 0
+; ACCEPT-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> poison, i32 [[TMP5]], i64 0
+; ACCEPT-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> poison, <4 x i32> zeroinitializer
+; ACCEPT-NEXT: [[TMP6:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
+; ACCEPT-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[BROADCAST_SPLAT]], ptr align 4 [[TMP6]], <4 x i1> [[TMP1]])
+; ACCEPT-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; ACCEPT-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; ACCEPT-NEXT: br i1 [[TMP7]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; ACCEPT: [[MIDDLE_BLOCK]]:
+; ACCEPT-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; ACCEPT-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; ACCEPT: [[SCALAR_PH]]:
+; ACCEPT-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; ACCEPT-NEXT: br label %[[FOR_BODY:.*]]
+; ACCEPT: [[FOR_BODY]]:
+; ACCEPT-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; ACCEPT-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; ACCEPT-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; ACCEPT-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; ACCEPT-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; ACCEPT: [[IF_THEN]]:
+; ACCEPT-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; ACCEPT-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; ACCEPT-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; ACCEPT-NEXT: br label %[[FOR_INC]]
+; ACCEPT: [[FOR_INC]]:
+; ACCEPT-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; ACCEPT-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; ACCEPT-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; ACCEPT: [[FOR_END]]:
+; ACCEPT-NEXT: ret void
+;
+; NOMASK-LABEL: define void @cond_invar_load_gate(
+; NOMASK-SAME: ptr noalias [[A:%.*]], ptr noalias [[COND:%.*]], ptr noalias [[INV:%.*]], i64 [[N:%.*]]) {
+; NOMASK-NEXT: [[ENTRY:.*]]:
+; NOMASK-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; NOMASK-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; NOMASK: [[VECTOR_PH]]:
+; NOMASK-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; NOMASK-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; NOMASK-NEXT: br label %[[VECTOR_BODY:.*]]
+; NOMASK: [[VECTOR_BODY]]:
+; NOMASK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[PRED_STORE_CONTINUE6:.*]] ]
+; NOMASK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; NOMASK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; NOMASK-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; NOMASK-NEXT: [[TMP2:%.*]] = extractelement <4 x i1> [[TMP1]], i64 0
+; NOMASK-NEXT: br i1 [[TMP2]], label %[[PRED_STORE_IF:.*]], label %[[PRED_STORE_CONTINUE:.*]]
+; NOMASK: [[PRED_STORE_IF]]:
+; NOMASK-NEXT: [[TMP3:%.*]] = load i32, ptr [[INV]], align 4
+; NOMASK-NEXT: [[TMP4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[INDEX]]
+; NOMASK-NEXT: store i32 [[TMP3]], ptr [[TMP4]], align 4
+; NOMASK-NEXT: br label %[[PRED_STORE_CONTINUE]]
+; NOMASK: [[PRED_STORE_CONTINUE]]:
+; NOMASK-NEXT: [[TMP5:%.*]] = extractelement <4 x i1> [[TMP1]], i64 1
+; NOMASK-NEXT: br i1 [[TMP5]], label %[[PRED_STORE_IF1:.*]], label %[[PRED_STORE_CONTINUE2:.*]]
+; NOMASK: [[PRED_STORE_IF1]]:
+; NOMASK-NEXT: [[TMP6:%.*]] = load i32, ptr [[INV]], align 4
+; NOMASK-NEXT: [[TMP7:%.*]] = add i64 [[INDEX]], 1
+; NOMASK-NEXT: [[TMP8:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP7]]
+; NOMASK-NEXT: store i32 [[TMP6]], ptr [[TMP8]], align 4
+; NOMASK-NEXT: br label %[[PRED_STORE_CONTINUE2]]
+; NOMASK: [[PRED_STORE_CONTINUE2]]:
+; NOMASK-NEXT: [[TMP9:%.*]] = extractelement <4 x i1> [[TMP1]], i64 2
+; NOMASK-NEXT: br i1 [[TMP9]], label %[[PRED_STORE_IF3:.*]], label %[[PRED_STORE_CONTINUE4:.*]]
+; NOMASK: [[PRED_STORE_IF3]]:
+; NOMASK-NEXT: [[TMP10:%.*]] = load i32, ptr [[INV]], align 4
+; NOMASK-NEXT: [[TMP11:%.*]] = add i64 [[INDEX]], 2
+; NOMASK-NEXT: [[TMP12:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP11]]
+; NOMASK-NEXT: store i32 [[TMP10]], ptr [[TMP12]], align 4
+; NOMASK-NEXT: br label %[[PRED_STORE_CONTINUE4]]
+; NOMASK: [[PRED_STORE_CONTINUE4]]:
+; NOMASK-NEXT: [[TMP13:%.*]] = extractelement <4 x i1> [[TMP1]], i64 3
+; NOMASK-NEXT: br i1 [[TMP13]], label %[[PRED_STORE_IF5:.*]], label %[[PRED_STORE_CONTINUE6]]
+; NOMASK: [[PRED_STORE_IF5]]:
+; NOMASK-NEXT: [[TMP14:%.*]] = load i32, ptr [[INV]], align 4
+; NOMASK-NEXT: [[TMP15:%.*]] = add i64 [[INDEX]], 3
+; NOMASK-NEXT: [[TMP16:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[TMP15]]
+; NOMASK-NEXT: store i32 [[TMP14]], ptr [[TMP16]], align 4
+; NOMASK-NEXT: br label %[[PRED_STORE_CONTINUE6]]
+; NOMASK: [[PRED_STORE_CONTINUE6]]:
+; NOMASK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; NOMASK-NEXT: [[TMP17:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; NOMASK-NEXT: br i1 [[TMP17]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; NOMASK: [[MIDDLE_BLOCK]]:
+; NOMASK-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; NOMASK-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; NOMASK: [[SCALAR_PH]]:
+; NOMASK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; NOMASK-NEXT: br label %[[FOR_BODY:.*]]
+; NOMASK: [[FOR_BODY]]:
+; NOMASK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; NOMASK-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; NOMASK-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; NOMASK-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; NOMASK-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; NOMASK: [[IF_THEN]]:
+; NOMASK-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; NOMASK-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; NOMASK-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; NOMASK-NEXT: br label %[[FOR_INC]]
+; NOMASK: [[FOR_INC]]:
+; NOMASK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; NOMASK-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; NOMASK-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; NOMASK: [[FOR_END]]:
+; NOMASK-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
+ %cv = load i32, ptr %cond.gep, align 4
+ %tobool = icmp ne i32 %cv, 0
+ br i1 %tobool, label %if.then, label %for.inc
+
+if.then:
+ %v = load i32, ptr %inv, align 4
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %v, ptr %a.gep, align 4
+ br label %for.inc
+
+for.inc:
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+;.
+; REJECT: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; REJECT: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; REJECT: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; REJECT: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+;.
+; ACCEPT: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; ACCEPT: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; ACCEPT: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; ACCEPT: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+;.
+; NOMASK: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; NOMASK: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; NOMASK: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; NOMASK: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+;.
diff --git a/llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll b/llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll
new file mode 100644
index 0000000000000..b3e45f36a7d86
--- /dev/null
+++ b/llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll
@@ -0,0 +1,526 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
+; RUN: -force-target-supports-masked-memory-ops -S | FileCheck %s --check-prefixes=CHECK,OFF
+; RUN: opt < %s -passes=loop-vectorize -force-vector-width=4 -force-vector-interleave=1 \
+; RUN: -force-target-supports-masked-memory-ops -enable-masked-invariant-load -S \
+; RUN: | FileCheck %s --check-prefixes=CHECK,ON
+
+;; Conditional load from a loop-invariant pointer.
+;;
+;; for (i = 0; i < n; ++i)
+;; if (cond[i])
+;; a[i] = *inv;
+;;
+;; The pointer %inv is loop-invariant and the load is predicated. Without
+;; -enable-masked-invariant-load the load is scalarised into a per-lane
+;; pred.load.if/continue chain. With the flag, widenPredicatedInvariantLoads
+;; rewrites it to an any-active-lane reduction over the mask, a single-lane
+;; masked load from %inv, and a broadcast of lane 0 across the vector.
+define void @cond_invar_load(ptr noalias %a, ptr noalias %cond, ptr noalias %inv, i64 %n) {
+; OFF-LABEL: define void @cond_invar_load(
+; OFF-SAME: ptr noalias [[A:%.*]], ptr noalias [[COND:%.*]], ptr noalias [[INV:%.*]], i64 [[N:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; OFF-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; OFF: [[VECTOR_PH]]:
+; OFF-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; OFF-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; OFF-NEXT: br label %[[VECTOR_BODY:.*]]
+; OFF: [[VECTOR_BODY]]:
+; OFF-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[PRED_LOAD_CONTINUE6:.*]] ]
+; OFF-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; OFF-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; OFF-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; OFF-NEXT: [[TMP2:%.*]] = extractelement <4 x i1> [[TMP1]], i64 0
+; OFF-NEXT: br i1 [[TMP2]], label %[[PRED_LOAD_IF:.*]], label %[[PRED_LOAD_CONTINUE:.*]]
+; OFF: [[PRED_LOAD_IF]]:
+; OFF-NEXT: [[TMP3:%.*]] = load i32, ptr [[INV]], align 4
+; OFF-NEXT: [[TMP4:%.*]] = insertelement <4 x i32> poison, i32 [[TMP3]], i64 0
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE]]
+; OFF: [[PRED_LOAD_CONTINUE]]:
+; OFF-NEXT: [[TMP5:%.*]] = phi <4 x i32> [ poison, %[[VECTOR_BODY]] ], [ [[TMP4]], %[[PRED_LOAD_IF]] ]
+; OFF-NEXT: [[TMP6:%.*]] = extractelement <4 x i1> [[TMP1]], i64 1
+; OFF-NEXT: br i1 [[TMP6]], label %[[PRED_LOAD_IF1:.*]], label %[[PRED_LOAD_CONTINUE2:.*]]
+; OFF: [[PRED_LOAD_IF1]]:
+; OFF-NEXT: [[TMP7:%.*]] = load i32, ptr [[INV]], align 4
+; OFF-NEXT: [[TMP8:%.*]] = insertelement <4 x i32> [[TMP5]], i32 [[TMP7]], i64 1
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE2]]
+; OFF: [[PRED_LOAD_CONTINUE2]]:
+; OFF-NEXT: [[TMP9:%.*]] = phi <4 x i32> [ [[TMP5]], %[[PRED_LOAD_CONTINUE]] ], [ [[TMP8]], %[[PRED_LOAD_IF1]] ]
+; OFF-NEXT: [[TMP10:%.*]] = extractelement <4 x i1> [[TMP1]], i64 2
+; OFF-NEXT: br i1 [[TMP10]], label %[[PRED_LOAD_IF3:.*]], label %[[PRED_LOAD_CONTINUE4:.*]]
+; OFF: [[PRED_LOAD_IF3]]:
+; OFF-NEXT: [[TMP11:%.*]] = load i32, ptr [[INV]], align 4
+; OFF-NEXT: [[TMP12:%.*]] = insertelement <4 x i32> [[TMP9]], i32 [[TMP11]], i64 2
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE4]]
+; OFF: [[PRED_LOAD_CONTINUE4]]:
+; OFF-NEXT: [[TMP13:%.*]] = phi <4 x i32> [ [[TMP9]], %[[PRED_LOAD_CONTINUE2]] ], [ [[TMP12]], %[[PRED_LOAD_IF3]] ]
+; OFF-NEXT: [[TMP14:%.*]] = extractelement <4 x i1> [[TMP1]], i64 3
+; OFF-NEXT: br i1 [[TMP14]], label %[[PRED_LOAD_IF5:.*]], label %[[PRED_LOAD_CONTINUE6]]
+; OFF: [[PRED_LOAD_IF5]]:
+; OFF-NEXT: [[TMP15:%.*]] = load i32, ptr [[INV]], align 4
+; OFF-NEXT: [[TMP16:%.*]] = insertelement <4 x i32> [[TMP13]], i32 [[TMP15]], i64 3
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE6]]
+; OFF: [[PRED_LOAD_CONTINUE6]]:
+; OFF-NEXT: [[TMP17:%.*]] = phi <4 x i32> [ [[TMP13]], %[[PRED_LOAD_CONTINUE4]] ], [ [[TMP16]], %[[PRED_LOAD_IF5]] ]
+; OFF-NEXT: [[TMP18:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
+; OFF-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[TMP17]], ptr align 4 [[TMP18]], <4 x i1> [[TMP1]])
+; OFF-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; OFF-NEXT: [[TMP19:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; OFF-NEXT: br i1 [[TMP19]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; OFF: [[MIDDLE_BLOCK]]:
+; OFF-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; OFF-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; OFF: [[SCALAR_PH]]:
+; OFF-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; OFF-NEXT: br label %[[FOR_BODY:.*]]
+; OFF: [[FOR_BODY]]:
+; OFF-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; OFF-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; OFF-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; OFF-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; OFF-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; OFF: [[IF_THEN]]:
+; OFF-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; OFF-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; OFF-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; OFF-NEXT: br label %[[FOR_INC]]
+; OFF: [[FOR_INC]]:
+; OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; OFF-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; OFF-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; OFF: [[FOR_END]]:
+; OFF-NEXT: ret void
+;
+; ON-LABEL: define void @cond_invar_load(
+; ON-SAME: ptr noalias [[A:%.*]], ptr noalias [[COND:%.*]], ptr noalias [[INV:%.*]], i64 [[N:%.*]]) {
+; ON-NEXT: [[ENTRY:.*]]:
+; ON-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; ON-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; ON: [[VECTOR_PH]]:
+; ON-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; ON-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; ON-NEXT: br label %[[VECTOR_BODY:.*]]
+; ON: [[VECTOR_BODY]]:
+; ON-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; ON-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; ON-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; ON-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; ON-NEXT: [[TMP2:%.*]] = freeze <4 x i1> [[TMP1]]
+; ON-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; ON-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
+; ON-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV]], <4 x i1> [[TMP4]], <4 x i32> poison)
+; ON-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[WIDE_MASKED_LOAD]], i64 0
+; ON-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> poison, i32 [[TMP5]], i64 0
+; ON-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> poison, <4 x i32> zeroinitializer
+; ON-NEXT: [[TMP6:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
+; ON-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[BROADCAST_SPLAT]], ptr align 4 [[TMP6]], <4 x i1> [[TMP1]])
+; ON-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; ON-NEXT: [[TMP7:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; ON-NEXT: br i1 [[TMP7]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP0:![0-9]+]]
+; ON: [[MIDDLE_BLOCK]]:
+; ON-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; ON-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; ON: [[SCALAR_PH]]:
+; ON-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; ON-NEXT: br label %[[FOR_BODY:.*]]
+; ON: [[FOR_BODY]]:
+; ON-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; ON-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; ON-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; ON-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; ON-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; ON: [[IF_THEN]]:
+; ON-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; ON-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; ON-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; ON-NEXT: br label %[[FOR_INC]]
+; ON: [[FOR_INC]]:
+; ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; ON-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; ON-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP3:![0-9]+]]
+; ON: [[FOR_END]]:
+; ON-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
+ %cv = load i32, ptr %cond.gep, align 4
+ %tobool = icmp ne i32 %cv, 0
+ br i1 %tobool, label %if.then, label %for.inc
+
+if.then:
+ %v = load i32, ptr %inv, align 4
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %v, ptr %a.gep, align 4
+ br label %for.inc
+
+for.inc:
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+;; Two distinct loop-invariant pointers loaded under the same predicate in the
+;; same block. Each load is independently rewritten to a single-lane masked
+;; load + broadcast, but they share the same any-of mask and lane-0 mask
+;; material thanks to the VPlan CSE that runs before unrolling.
+;;
+;; for (i = 0; i < n; ++i)
+;; if (cond[i])
+;; a[i] = *inv1 + *inv2;
+define void @cond_invar_load_two_ptrs(ptr noalias %a, ptr noalias %cond, ptr noalias %inv1, ptr noalias %inv2, i64 %n) {
+; OFF-LABEL: define void @cond_invar_load_two_ptrs(
+; OFF-SAME: ptr noalias [[A:%.*]], ptr noalias [[COND:%.*]], ptr noalias [[INV1:%.*]], ptr noalias [[INV2:%.*]], i64 [[N:%.*]]) {
+; OFF-NEXT: [[ENTRY:.*]]:
+; OFF-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; OFF-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; OFF: [[VECTOR_PH]]:
+; OFF-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; OFF-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; OFF-NEXT: br label %[[VECTOR_BODY:.*]]
+; OFF: [[VECTOR_BODY]]:
+; OFF-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[PRED_LOAD_CONTINUE6:.*]] ]
+; OFF-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; OFF-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; OFF-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; OFF-NEXT: [[TMP2:%.*]] = extractelement <4 x i1> [[TMP1]], i64 0
+; OFF-NEXT: br i1 [[TMP2]], label %[[PRED_LOAD_IF:.*]], label %[[PRED_LOAD_CONTINUE:.*]]
+; OFF: [[PRED_LOAD_IF]]:
+; OFF-NEXT: [[TMP3:%.*]] = load i32, ptr [[INV1]], align 4
+; OFF-NEXT: [[TMP4:%.*]] = insertelement <4 x i32> poison, i32 [[TMP3]], i64 0
+; OFF-NEXT: [[TMP5:%.*]] = load i32, ptr [[INV2]], align 4
+; OFF-NEXT: [[TMP6:%.*]] = insertelement <4 x i32> poison, i32 [[TMP5]], i64 0
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE]]
+; OFF: [[PRED_LOAD_CONTINUE]]:
+; OFF-NEXT: [[TMP7:%.*]] = phi <4 x i32> [ poison, %[[VECTOR_BODY]] ], [ [[TMP4]], %[[PRED_LOAD_IF]] ]
+; OFF-NEXT: [[TMP8:%.*]] = phi <4 x i32> [ poison, %[[VECTOR_BODY]] ], [ [[TMP6]], %[[PRED_LOAD_IF]] ]
+; OFF-NEXT: [[TMP9:%.*]] = extractelement <4 x i1> [[TMP1]], i64 1
+; OFF-NEXT: br i1 [[TMP9]], label %[[PRED_LOAD_IF1:.*]], label %[[PRED_LOAD_CONTINUE2:.*]]
+; OFF: [[PRED_LOAD_IF1]]:
+; OFF-NEXT: [[TMP10:%.*]] = load i32, ptr [[INV1]], align 4
+; OFF-NEXT: [[TMP11:%.*]] = insertelement <4 x i32> [[TMP7]], i32 [[TMP10]], i64 1
+; OFF-NEXT: [[TMP12:%.*]] = load i32, ptr [[INV2]], align 4
+; OFF-NEXT: [[TMP13:%.*]] = insertelement <4 x i32> [[TMP8]], i32 [[TMP12]], i64 1
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE2]]
+; OFF: [[PRED_LOAD_CONTINUE2]]:
+; OFF-NEXT: [[TMP14:%.*]] = phi <4 x i32> [ [[TMP7]], %[[PRED_LOAD_CONTINUE]] ], [ [[TMP11]], %[[PRED_LOAD_IF1]] ]
+; OFF-NEXT: [[TMP15:%.*]] = phi <4 x i32> [ [[TMP8]], %[[PRED_LOAD_CONTINUE]] ], [ [[TMP13]], %[[PRED_LOAD_IF1]] ]
+; OFF-NEXT: [[TMP16:%.*]] = extractelement <4 x i1> [[TMP1]], i64 2
+; OFF-NEXT: br i1 [[TMP16]], label %[[PRED_LOAD_IF3:.*]], label %[[PRED_LOAD_CONTINUE4:.*]]
+; OFF: [[PRED_LOAD_IF3]]:
+; OFF-NEXT: [[TMP17:%.*]] = load i32, ptr [[INV1]], align 4
+; OFF-NEXT: [[TMP18:%.*]] = insertelement <4 x i32> [[TMP14]], i32 [[TMP17]], i64 2
+; OFF-NEXT: [[TMP19:%.*]] = load i32, ptr [[INV2]], align 4
+; OFF-NEXT: [[TMP20:%.*]] = insertelement <4 x i32> [[TMP15]], i32 [[TMP19]], i64 2
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE4]]
+; OFF: [[PRED_LOAD_CONTINUE4]]:
+; OFF-NEXT: [[TMP21:%.*]] = phi <4 x i32> [ [[TMP14]], %[[PRED_LOAD_CONTINUE2]] ], [ [[TMP18]], %[[PRED_LOAD_IF3]] ]
+; OFF-NEXT: [[TMP22:%.*]] = phi <4 x i32> [ [[TMP15]], %[[PRED_LOAD_CONTINUE2]] ], [ [[TMP20]], %[[PRED_LOAD_IF3]] ]
+; OFF-NEXT: [[TMP23:%.*]] = extractelement <4 x i1> [[TMP1]], i64 3
+; OFF-NEXT: br i1 [[TMP23]], label %[[PRED_LOAD_IF5:.*]], label %[[PRED_LOAD_CONTINUE6]]
+; OFF: [[PRED_LOAD_IF5]]:
+; OFF-NEXT: [[TMP24:%.*]] = load i32, ptr [[INV1]], align 4
+; OFF-NEXT: [[TMP25:%.*]] = insertelement <4 x i32> [[TMP21]], i32 [[TMP24]], i64 3
+; OFF-NEXT: [[TMP26:%.*]] = load i32, ptr [[INV2]], align 4
+; OFF-NEXT: [[TMP27:%.*]] = insertelement <4 x i32> [[TMP22]], i32 [[TMP26]], i64 3
+; OFF-NEXT: br label %[[PRED_LOAD_CONTINUE6]]
+; OFF: [[PRED_LOAD_CONTINUE6]]:
+; OFF-NEXT: [[TMP28:%.*]] = phi <4 x i32> [ [[TMP21]], %[[PRED_LOAD_CONTINUE4]] ], [ [[TMP25]], %[[PRED_LOAD_IF5]] ]
+; OFF-NEXT: [[TMP29:%.*]] = phi <4 x i32> [ [[TMP22]], %[[PRED_LOAD_CONTINUE4]] ], [ [[TMP27]], %[[PRED_LOAD_IF5]] ]
+; OFF-NEXT: [[TMP30:%.*]] = add <4 x i32> [[TMP28]], [[TMP29]]
+; OFF-NEXT: [[TMP31:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
+; OFF-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[TMP30]], ptr align 4 [[TMP31]], <4 x i1> [[TMP1]])
+; OFF-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; OFF-NEXT: [[TMP32:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; OFF-NEXT: br i1 [[TMP32]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
+; OFF: [[MIDDLE_BLOCK]]:
+; OFF-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; OFF-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; OFF: [[SCALAR_PH]]:
+; OFF-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; OFF-NEXT: br label %[[FOR_BODY:.*]]
+; OFF: [[FOR_BODY]]:
+; OFF-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; OFF-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; OFF-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; OFF-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; OFF-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; OFF: [[IF_THEN]]:
+; OFF-NEXT: [[V1:%.*]] = load i32, ptr [[INV1]], align 4
+; OFF-NEXT: [[V2:%.*]] = load i32, ptr [[INV2]], align 4
+; OFF-NEXT: [[SUM:%.*]] = add i32 [[V1]], [[V2]]
+; OFF-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; OFF-NEXT: store i32 [[SUM]], ptr [[A_GEP]], align 4
+; OFF-NEXT: br label %[[FOR_INC]]
+; OFF: [[FOR_INC]]:
+; OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; OFF-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; OFF-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; OFF: [[FOR_END]]:
+; OFF-NEXT: ret void
+;
+; ON-LABEL: define void @cond_invar_load_two_ptrs(
+; ON-SAME: ptr noalias [[A:%.*]], ptr noalias [[COND:%.*]], ptr noalias [[INV1:%.*]], ptr noalias [[INV2:%.*]], i64 [[N:%.*]]) {
+; ON-NEXT: [[ENTRY:.*]]:
+; ON-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; ON-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; ON: [[VECTOR_PH]]:
+; ON-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; ON-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; ON-NEXT: br label %[[VECTOR_BODY:.*]]
+; ON: [[VECTOR_BODY]]:
+; ON-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; ON-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; ON-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; ON-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; ON-NEXT: [[TMP2:%.*]] = freeze <4 x i1> [[TMP1]]
+; ON-NEXT: [[TMP3:%.*]] = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> [[TMP2]])
+; ON-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
+; ON-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV1]], <4 x i1> [[TMP4]], <4 x i32> poison)
+; ON-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[WIDE_MASKED_LOAD]], i64 0
+; ON-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> poison, i32 [[TMP5]], i64 0
+; ON-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> poison, <4 x i32> zeroinitializer
+; ON-NEXT: [[WIDE_MASKED_LOAD1:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV2]], <4 x i1> [[TMP4]], <4 x i32> poison)
+; ON-NEXT: [[TMP6:%.*]] = extractelement <4 x i32> [[WIDE_MASKED_LOAD1]], i64 0
+; ON-NEXT: [[BROADCAST_SPLATINSERT2:%.*]] = insertelement <4 x i32> poison, i32 [[TMP6]], i64 0
+; ON-NEXT: [[BROADCAST_SPLAT3:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT2]], <4 x i32> poison, <4 x i32> zeroinitializer
+; ON-NEXT: [[TMP7:%.*]] = extractelement <4 x i32> [[BROADCAST_SPLAT]], i64 0
+; ON-NEXT: [[TMP8:%.*]] = extractelement <4 x i32> [[BROADCAST_SPLAT3]], i64 0
+; ON-NEXT: [[TMP9:%.*]] = add i32 [[TMP7]], [[TMP8]]
+; ON-NEXT: [[BROADCAST_SPLATINSERT4:%.*]] = insertelement <4 x i32> poison, i32 [[TMP9]], i64 0
+; ON-NEXT: [[BROADCAST_SPLAT5:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT4]], <4 x i32> poison, <4 x i32> zeroinitializer
+; ON-NEXT: [[TMP10:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
+; ON-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[BROADCAST_SPLAT5]], ptr align 4 [[TMP10]], <4 x i1> [[TMP1]])
+; ON-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; ON-NEXT: [[TMP11:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; ON-NEXT: br i1 [[TMP11]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP4:![0-9]+]]
+; ON: [[MIDDLE_BLOCK]]:
+; ON-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; ON-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; ON: [[SCALAR_PH]]:
+; ON-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; ON-NEXT: br label %[[FOR_BODY:.*]]
+; ON: [[FOR_BODY]]:
+; ON-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; ON-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; ON-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; ON-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; ON-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; ON: [[IF_THEN]]:
+; ON-NEXT: [[V1:%.*]] = load i32, ptr [[INV1]], align 4
+; ON-NEXT: [[V2:%.*]] = load i32, ptr [[INV2]], align 4
+; ON-NEXT: [[SUM:%.*]] = add i32 [[V1]], [[V2]]
+; ON-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; ON-NEXT: store i32 [[SUM]], ptr [[A_GEP]], align 4
+; ON-NEXT: br label %[[FOR_INC]]
+; ON: [[FOR_INC]]:
+; ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; ON-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; ON-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP5:![0-9]+]]
+; ON: [[FOR_END]]:
+; ON-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
+ %cv = load i32, ptr %cond.gep, align 4
+ %tobool = icmp ne i32 %cv, 0
+ br i1 %tobool, label %if.then, label %for.inc
+
+if.then:
+ %v1 = load i32, ptr %inv1, align 4
+ %v2 = load i32, ptr %inv2, align 4
+ %sum = add i32 %v1, %v2
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %sum, ptr %a.gep, align 4
+ br label %for.inc
+
+for.inc:
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+;; Negative test: the conditional load's pointer (%src + i) is loop-VARIANT, so
+;; even with -enable-masked-invariant-load the transform does not fire and the
+;; load stays a regular consecutive masked load; no any-of reduction or
+;; broadcast is produced.
+;;
+;; for (i = 0; i < n; ++i)
+;; if (cond[i])
+;; a[i] = src[i];
+define void @cond_load_not_invariant(ptr noalias %a, ptr noalias %src, ptr noalias %cond, i64 %n) {
+; CHECK-LABEL: define void @cond_load_not_invariant(
+; CHECK-SAME: ptr noalias [[A:%.*]], ptr noalias [[SRC:%.*]], ptr noalias [[COND:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; CHECK-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; CHECK: [[VECTOR_PH]]:
+; CHECK-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; CHECK-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; CHECK-NEXT: br label %[[VECTOR_BODY:.*]]
+; CHECK: [[VECTOR_BODY]]:
+; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; CHECK-NEXT: [[TMP0:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[INDEX]]
+; CHECK-NEXT: [[WIDE_LOAD:%.*]] = load <4 x i32>, ptr [[TMP0]], align 4
+; CHECK-NEXT: [[TMP1:%.*]] = icmp ne <4 x i32> [[WIDE_LOAD]], zeroinitializer
+; CHECK-NEXT: [[TMP2:%.*]] = getelementptr i32, ptr [[SRC]], i64 [[INDEX]]
+; CHECK-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[TMP2]], <4 x i1> [[TMP1]], <4 x i32> poison)
+; CHECK-NEXT: [[TMP3:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
+; CHECK-NEXT: call void @llvm.masked.store.v4i32.p0(<4 x i32> [[WIDE_MASKED_LOAD]], ptr align 4 [[TMP3]], <4 x i1> [[TMP1]])
+; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT: [[TMP4:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[TMP4]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP6:![0-9]+]]
+; CHECK: [[MIDDLE_BLOCK]]:
+; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; CHECK: [[SCALAR_PH]]:
+; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; CHECK-NEXT: br label %[[FOR_BODY:.*]]
+; CHECK: [[FOR_BODY]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_INC:.*]] ]
+; CHECK-NEXT: [[COND_GEP:%.*]] = getelementptr inbounds i32, ptr [[COND]], i64 [[I]]
+; CHECK-NEXT: [[CV:%.*]] = load i32, ptr [[COND_GEP]], align 4
+; CHECK-NEXT: [[TOBOOL:%.*]] = icmp ne i32 [[CV]], 0
+; CHECK-NEXT: br i1 [[TOBOOL]], label %[[IF_THEN:.*]], label %[[FOR_INC]]
+; CHECK: [[IF_THEN]]:
+; CHECK-NEXT: [[SRC_GEP:%.*]] = getelementptr inbounds i32, ptr [[SRC]], i64 [[I]]
+; CHECK-NEXT: [[V:%.*]] = load i32, ptr [[SRC_GEP]], align 4
+; CHECK-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store i32 [[V]], ptr [[A_GEP]], align 4
+; CHECK-NEXT: br label %[[FOR_INC]]
+; CHECK: [[FOR_INC]]:
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; CHECK-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP7:![0-9]+]]
+; CHECK: [[FOR_END]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.inc ]
+ %cond.gep = getelementptr inbounds i32, ptr %cond, i64 %i
+ %cv = load i32, ptr %cond.gep, align 4
+ %tobool = icmp ne i32 %cv, 0
+ br i1 %tobool, label %if.then, label %for.inc
+
+if.then:
+ %src.gep = getelementptr inbounds i32, ptr %src, i64 %i
+ %v = load i32, ptr %src.gep, align 4
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %v, ptr %a.gep, align 4
+ br label %for.inc
+
+for.inc:
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+
+;; Negative test: the invariant load is UNCONDITIONAL (not predicated), so
+;; widenPredicatedInvariantLoads skips it (it only targets predicated replicate
+;; loads). It is handled by the normal uniform-load machinery and stays a plain
+;; scalar load from %inv, regardless of -enable-masked-invariant-load.
+;;
+;; for (i = 0; i < n; ++i)
+;; a[i] = *inv + i;
+define void @uncond_invar_load(ptr noalias %a, ptr noalias %inv, i64 %n) {
+; CHECK-LABEL: define void @uncond_invar_load(
+; CHECK-SAME: ptr noalias [[A:%.*]], ptr noalias [[INV:%.*]], i64 [[N:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: [[MIN_ITERS_CHECK:%.*]] = icmp ult i64 [[N]], 4
+; CHECK-NEXT: br i1 [[MIN_ITERS_CHECK]], label %[[SCALAR_PH:.*]], label %[[VECTOR_PH:.*]]
+; CHECK: [[VECTOR_PH]]:
+; CHECK-NEXT: [[N_MOD_VF:%.*]] = urem i64 [[N]], 4
+; CHECK-NEXT: [[N_VEC:%.*]] = sub i64 [[N]], [[N_MOD_VF]]
+; CHECK-NEXT: br label %[[VECTOR_BODY:.*]]
+; CHECK: [[VECTOR_BODY]]:
+; CHECK-NEXT: [[INDEX:%.*]] = phi i64 [ 0, %[[VECTOR_PH]] ], [ [[INDEX_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; CHECK-NEXT: [[VEC_IND:%.*]] = phi <4 x i32> [ <i32 0, i32 1, i32 2, i32 3>, %[[VECTOR_PH]] ], [ [[VEC_IND_NEXT:%.*]], %[[VECTOR_BODY]] ]
+; CHECK-NEXT: [[TMP0:%.*]] = load i32, ptr [[INV]], align 4
+; CHECK-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> poison, i32 [[TMP0]], i64 0
+; CHECK-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT: [[TMP1:%.*]] = add <4 x i32> [[BROADCAST_SPLAT]], [[VEC_IND]]
+; CHECK-NEXT: [[TMP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[INDEX]]
+; CHECK-NEXT: store <4 x i32> [[TMP1]], ptr [[TMP2]], align 4
+; CHECK-NEXT: [[INDEX_NEXT]] = add nuw i64 [[INDEX]], 4
+; CHECK-NEXT: [[VEC_IND_NEXT]] = add <4 x i32> [[VEC_IND]], splat (i32 4)
+; CHECK-NEXT: [[TMP3:%.*]] = icmp eq i64 [[INDEX_NEXT]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[TMP3]], label %[[MIDDLE_BLOCK:.*]], label %[[VECTOR_BODY]], !llvm.loop [[LOOP8:![0-9]+]]
+; CHECK: [[MIDDLE_BLOCK]]:
+; CHECK-NEXT: [[CMP_N:%.*]] = icmp eq i64 [[N]], [[N_VEC]]
+; CHECK-NEXT: br i1 [[CMP_N]], label %[[FOR_END:.*]], label %[[SCALAR_PH]]
+; CHECK: [[SCALAR_PH]]:
+; CHECK-NEXT: [[BC_RESUME_VAL:%.*]] = phi i64 [ [[N_VEC]], %[[MIDDLE_BLOCK]] ], [ 0, %[[ENTRY]] ]
+; CHECK-NEXT: br label %[[FOR_BODY:.*]]
+; CHECK: [[FOR_BODY]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ [[BC_RESUME_VAL]], %[[SCALAR_PH]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[V:%.*]] = load i32, ptr [[INV]], align 4
+; CHECK-NEXT: [[IDX:%.*]] = trunc i64 [[I]] to i32
+; CHECK-NEXT: [[ADD:%.*]] = add i32 [[V]], [[IDX]]
+; CHECK-NEXT: [[A_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store i32 [[ADD]], ptr [[A_GEP]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 1
+; CHECK-NEXT: [[EXITCOND:%.*]] = icmp eq i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[EXITCOND]], label %[[FOR_END]], label %[[FOR_BODY]], !llvm.loop [[LOOP9:![0-9]+]]
+; CHECK: [[FOR_END]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
+ %v = load i32, ptr %inv, align 4
+ %idx = trunc i64 %i to i32
+ %add = add i32 %v, %idx
+ %a.gep = getelementptr inbounds i32, ptr %a, i64 %i
+ store i32 %add, ptr %a.gep, align 4
+ %i.next = add nuw nsw i64 %i, 1
+ %exitcond = icmp eq i64 %i.next, %n
+ br i1 %exitcond, label %for.end, label %for.body
+
+for.end:
+ ret void
+}
+;.
+; OFF: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; OFF: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; OFF: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; OFF: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+; OFF: [[LOOP4]] = distinct !{[[LOOP4]], [[META1]], [[META2]]}
+; OFF: [[LOOP5]] = distinct !{[[LOOP5]], [[META2]], [[META1]]}
+; OFF: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]]}
+; OFF: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META1]]}
+; OFF: [[LOOP8]] = distinct !{[[LOOP8]], [[META1]], [[META2]]}
+; OFF: [[LOOP9]] = distinct !{[[LOOP9]], [[META2]], [[META1]]}
+;.
+; ON: [[LOOP0]] = distinct !{[[LOOP0]], [[META1:![0-9]+]], [[META2:![0-9]+]]}
+; ON: [[META1]] = !{!"llvm.loop.isvectorized", i32 1}
+; ON: [[META2]] = !{!"llvm.loop.unroll.runtime.disable"}
+; ON: [[LOOP3]] = distinct !{[[LOOP3]], [[META2]], [[META1]]}
+; ON: [[LOOP4]] = distinct !{[[LOOP4]], [[META1]], [[META2]]}
+; ON: [[LOOP5]] = distinct !{[[LOOP5]], [[META2]], [[META1]]}
+; ON: [[LOOP6]] = distinct !{[[LOOP6]], [[META1]], [[META2]]}
+; ON: [[LOOP7]] = distinct !{[[LOOP7]], [[META2]], [[META1]]}
+; ON: [[LOOP8]] = distinct !{[[LOOP8]], [[META1]], [[META2]]}
+; ON: [[LOOP9]] = distinct !{[[LOOP9]], [[META2]], [[META1]]}
+;.
>From 59673d1bdda7e4f2ee575d77734026d36f5e5b82 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Wed, 29 Jul 2026 18:23:05 +0530
Subject: [PATCH 4/5] [LV] Address review comments for
widenPredicatedInvariantLoads
- Extrended the costing by introducing shuffle/broadcast cost
---
llvm/lib/Transforms/Vectorize/VPlan.h | 8 ++++++++
llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp | 7 +++++++
llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp | 4 +++-
.../Transforms/LoopVectorize/masked-invariant-load.ll | 8 +-------
4 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.h b/llvm/lib/Transforms/Vectorize/VPlan.h
index a5fa558002c4e..f30fa265382e2 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.h
+++ b/llvm/lib/Transforms/Vectorize/VPlan.h
@@ -34,6 +34,7 @@
#include "llvm/ADT/ilist_node.h"
#include "llvm/Analysis/IVDescriptors.h"
#include "llvm/Analysis/MemoryLocation.h"
+#include "llvm/Analysis/TargetTransformInfo.h"
#include "llvm/Analysis/VectorUtils.h"
#include "llvm/IR/DebugLoc.h"
#include "llvm/IR/FMF.h"
@@ -1445,6 +1446,13 @@ class LLVM_ABI_FOR_TEST VPInstruction : public VPRecipeWithIRFlags,
static InstructionCost computeExtractElementCost(VectorType *VecTy,
VPCostContext &Ctx);
+ /// Return the cost of a shuffle of kind \p Kind over vector type \p VecTy.
+ /// Static helper so callers can price the operation without constructing a
+ /// recipe.
+ static InstructionCost
+ computeShuffleCost(TargetTransformInfo::ShuffleKind Kind, VectorType *VecTy,
+ VPCostContext &Ctx);
+
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
/// Print the VPInstruction to dbgs() (for debugging).
LLVM_DUMP_METHOD void dump() const;
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index 68bee7dc7439a..9515e5b3d4037 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -1329,6 +1329,13 @@ InstructionCost VPInstruction::computeExtractElementCost(VectorType *VecTy,
Ctx.CostKind);
}
+InstructionCost
+VPInstruction::computeShuffleCost(TargetTransformInfo::ShuffleKind Kind,
+ VectorType *VecTy, VPCostContext &Ctx) {
+ return Ctx.TTI.getShuffleCost(Kind, VecTy, VecTy, /*Mask=*/{}, Ctx.CostKind,
+ /*Index=*/0);
+}
+
InstructionCost VPInstruction::computeCost(ElementCount VF,
VPCostContext &Ctx) const {
if (Instruction::isBinaryOp(getOpcode())) {
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index 3ed786d3343f5..86811e3e7c4c2 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -3906,7 +3906,9 @@ void VPlanTransforms::widenPredicatedInvariantLoads(VPlan &Plan,
VPInstruction::computeAnyOfCost(MaskTy, CostCtx) +
VPWidenLoadRecipe::computeMaskedCost(Instruction::Load, VecTy,
Alignment, AS, CostCtx) +
- VPInstruction::computeExtractElementCost(VecTy, CostCtx);
+ VPInstruction::computeExtractElementCost(VecTy, CostCtx) +
+ VPInstruction::computeShuffleCost(TargetTransformInfo::SK_Broadcast,
+ VecTy, CostCtx);
InstructionCost OldCost = RepR->computeCost(VF, CostCtx);
if (NewCost > OldCost) {
Beneficial = false;
diff --git a/llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll b/llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll
index b3e45f36a7d86..a04d53e7f5e17 100644
--- a/llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll
+++ b/llvm/test/Transforms/LoopVectorize/masked-invariant-load.ll
@@ -285,15 +285,9 @@ define void @cond_invar_load_two_ptrs(ptr noalias %a, ptr noalias %cond, ptr noa
; ON-NEXT: [[TMP4:%.*]] = insertelement <4 x i1> zeroinitializer, i1 [[TMP3]], i32 0
; ON-NEXT: [[WIDE_MASKED_LOAD:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV1]], <4 x i1> [[TMP4]], <4 x i32> poison)
; ON-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[WIDE_MASKED_LOAD]], i64 0
-; ON-NEXT: [[BROADCAST_SPLATINSERT:%.*]] = insertelement <4 x i32> poison, i32 [[TMP5]], i64 0
-; ON-NEXT: [[BROADCAST_SPLAT:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT]], <4 x i32> poison, <4 x i32> zeroinitializer
; ON-NEXT: [[WIDE_MASKED_LOAD1:%.*]] = call <4 x i32> @llvm.masked.load.v4i32.p0(ptr align 4 [[INV2]], <4 x i1> [[TMP4]], <4 x i32> poison)
; ON-NEXT: [[TMP6:%.*]] = extractelement <4 x i32> [[WIDE_MASKED_LOAD1]], i64 0
-; ON-NEXT: [[BROADCAST_SPLATINSERT2:%.*]] = insertelement <4 x i32> poison, i32 [[TMP6]], i64 0
-; ON-NEXT: [[BROADCAST_SPLAT3:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT2]], <4 x i32> poison, <4 x i32> zeroinitializer
-; ON-NEXT: [[TMP7:%.*]] = extractelement <4 x i32> [[BROADCAST_SPLAT]], i64 0
-; ON-NEXT: [[TMP8:%.*]] = extractelement <4 x i32> [[BROADCAST_SPLAT3]], i64 0
-; ON-NEXT: [[TMP9:%.*]] = add i32 [[TMP7]], [[TMP8]]
+; ON-NEXT: [[TMP9:%.*]] = add i32 [[TMP5]], [[TMP6]]
; ON-NEXT: [[BROADCAST_SPLATINSERT4:%.*]] = insertelement <4 x i32> poison, i32 [[TMP9]], i64 0
; ON-NEXT: [[BROADCAST_SPLAT5:%.*]] = shufflevector <4 x i32> [[BROADCAST_SPLATINSERT4]], <4 x i32> poison, <4 x i32> zeroinitializer
; ON-NEXT: [[TMP10:%.*]] = getelementptr i32, ptr [[A]], i64 [[INDEX]]
>From 8ae2c8f7989d7dbe422c7c5f2637671605912532 Mon Sep 17 00:00:00 2001
From: Ashutosh Nema <ashu1212 at gmail.com>
Date: Wed, 29 Jul 2026 18:26:04 +0530
Subject: [PATCH 5/5] Test update for widenPredicatedInvariantLoads
---
.../LoopVectorize/VPlan/vplan-print-before-after-all.ll | 1 +
1 file changed, 1 insertion(+)
diff --git a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll
index a7271524f0191..62ca1921defc3 100644
--- a/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll
+++ b/llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll
@@ -40,6 +40,7 @@
; CHECK: VPlan for loop in 'foo' [[BEFORE_OR_AFTER]] VPlanTransforms::dropPoisonGeneratingRecipes
; CHECK: VPlan for loop in 'foo' [[BEFORE_OR_AFTER]] VPlanTransforms::hoistPredicatedLoads
; CHECK: VPlan for loop in 'foo' [[BEFORE_OR_AFTER]] VPlanTransforms::sinkPredicatedStores
+; CHECK: VPlan for loop in 'foo' [[BEFORE_OR_AFTER]] VPlanTransforms::widenPredicatedInvariantLoads
; CHECK: VPlan for loop in 'foo' [[BEFORE_OR_AFTER]] VPlanTransforms::truncateToMinimalBitwidths
; CHECK-BEFORE: VPlan for loop in 'foo' [[BEFORE_OR_AFTER]] VPlanTransforms::optimize
; CHECK: VPlan for loop in 'foo' [[BEFORE_OR_AFTER]] removeRedundantInductionCasts
More information about the llvm-commits
mailing list