[llvm] [LV] Vectorize early exit loops with stores using masking (PR #178454)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Jan 28 08:03:53 PST 2026
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-vectorizers
Author: Graham Hunter (huntergr-arm)
<details>
<summary>Changes</summary>
This is an alternative approach to vectorizing early exit loops with stores (vs. #<!-- -->148626) that avoids needing to add an extra check block. I'll revisit the other patch later, but this is a fairly straightforward approach that should work on vector ISAs supporting masked memory ops.
The basic approach is to create a mask covering all lanes _before_ any exiting lane, using cttz.elts and active.lane.mask (which sets all lanes to true if the uncountable exit wasn't taken). If the uncountable exit was taken, then there will still be one scalar iteration left to perform after the vector loop, which will also handle which exit block we should branch to.
We no longer need to advance exit conditions in the vector body to the next iteration (compared to the other PR), though we still need to move the recipes needed to generate the exit condition (depending on which memory operations are first in the loop).
The advantage this has over a full in-loop mask approach is that we don't need to form intermediate masks for each uncountable exit; while I haven't tried to mix this with the ongoing multiple-exit work yet, we should be able to handle them without increasing the amount of generated per-exit code. We also won't need to unpick which exit condition was met first.
I've tried a little bit of benchmarking (using ACLE to create the loops, since the codegen for this is quite bad right now) on a Neoverse-V2 core, and at least for my simple example code (adding 1 to members of an array until a separate exit condition occurred) this was competitive performance-wise with full masking and bail-to-scalar for the entire last vector iteration.
---
Patch is 88.46 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/178454.diff
10 Files Affected:
- (modified) llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h (+6)
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp (+13-6)
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+47-3)
- (modified) llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp (+4-2)
- (modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp (+175-15)
- (modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.h (+10-3)
- (modified) llvm/lib/Transforms/Vectorize/VPlanUtils.cpp (+36-3)
- (added) llvm/test/Transforms/LoopVectorize/AArch64/early_exit_with_stores.ll (+1260)
- (modified) llvm/test/Transforms/LoopVectorize/early_exit_store_legality.ll (+9-9)
- (modified) llvm/unittests/Transforms/Vectorize/VPlanTestBase.h (+4-1)
``````````diff
diff --git a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
index f82fc588639dd..59cc98297c93c 100644
--- a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
+++ b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
@@ -483,6 +483,10 @@ class LoopVectorizationLegality {
return CountableExitingBlocks;
}
+ const ArrayRef<BasicBlock *> getUncountableExitingBlocks() const {
+ return UncountableExitingBlocksVec;
+ }
+
private:
/// Return true if the pre-header, exiting and latch blocks of \p Lp and all
/// its nested loops are considered legal for vectorization. These legal
@@ -738,6 +742,8 @@ class LoopVectorizationLegality {
/// the exact backedge taken count is not computable.
SmallVector<BasicBlock *, 4> CountableExitingBlocks;
+ SmallVector<BasicBlock *, 4> UncountableExitingBlocksVec;
+
/// True if the loop has uncountable early exits.
bool HasUncountableEarlyExit = false;
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index 1b000be130e49..978d91ea67338 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -1874,6 +1874,7 @@ bool LoopVectorizationLegality::isVectorizableEarlyExitLoop() {
<< *SymbolicMaxBTC << '\n');
HasUncountableEarlyExit = true;
UncountableExitWithSideEffects = HasSideEffects;
+ UncountableExitingBlocksVec.append(UncountableExitingBlocks);
return true;
}
@@ -2058,12 +2059,18 @@ bool LoopVectorizationLegality::canVectorize(bool UseVPlanNativePath) {
return false;
}
- // Bail out for state-changing loops with uncountable exits for now.
- if (UncountableExitWithSideEffects) {
- reportVectorizationFailure(
- "Writes to memory unsupported in early exit loops",
- "Cannot vectorize early exit loop with writes to memory",
- "WritesInEarlyExitLoop", ORE, TheLoop);
+ // TODO: Remove this restriction once we're sure it's safe to do so.
+ // If we're doing full masking we would need to store from the last
+ // active lane instead of the last lane according to VF. Doing the last
+ // iteration in the scalar loop should be ok, but we need to enforce it.
+ if (hasUncountableExitWithSideEffects() &&
+ !LAI->getStoresToInvariantAddresses().empty()) {
+ LLVM_DEBUG(dbgs() << "LV: Cannot vectorize early exit loops with stores to "
+ "loop-invariant addresses\n");
+ reportVectorizationFailure("Cannot vectorize EE loop with LI stores",
+ "Cannot vectorize early exit loops with stores "
+ "to loop-invariant addresses",
+ ORE, TheLoop);
return false;
}
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 7b6264771ef3a..357204b5b110c 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -399,6 +399,12 @@ static cl::opt<bool> EnableEarlyExitVectorization(
cl::desc(
"Enable vectorization of early exit loops with uncountable exits."));
+static cl::opt<bool> EnableEarlyExitWithSideEffectVectorization(
+ "enable-early-exit-with-side-effect-vectorization", cl::init(false),
+ cl::Hidden,
+ cl::desc("Enable vectorization of early exit loops with uncountable exits "
+ "and side effects"));
+
static cl::opt<bool> ConsiderRegPressure(
"vectorizer-consider-reg-pressure", cl::init(false), cl::Hidden,
cl::desc("Discard VFs if their register pressure is too high."));
@@ -7298,6 +7304,7 @@ VectorizationFactor LoopVectorizationPlanner::computeBestVF() {
});
assert(
(BestFactor.Width == LegacyVF.Width || BestPlan.hasEarlyExit() ||
+ Legal->hasUncountableExitWithSideEffects() ||
!Legal->getLAI()->getSymbolicStrides().empty() || UsesEVLGatherScatter ||
planContainsAdditionalSimplifications(
getPlanFor(BestFactor.Width), CostCtx, OrigLoop, BestFactor.Width) ||
@@ -8368,6 +8375,24 @@ void LoopVectorizationPlanner::buildVPlansWithVPRecipes(ElementCount MinVF,
*Plan, CM.getMaxSafeElements());
VPlanTransforms::runPass(VPlanTransforms::optimizeEVLMasks, *Plan);
}
+
+ // If we have side effects (e.g. stores) in a loop with uncountable
+ // exits, we must ensure the early exit condition is known before any
+ // state changing operations take place.
+ if (!Plan->hasScalarVFOnly() &&
+ Legal->hasUncountableExitWithSideEffects()) {
+ ArrayRef<BasicBlock *> UncountableExits =
+ Legal->getUncountableExitingBlocks();
+ // TODO: Support more than one.
+ if (UncountableExits.size() > 1)
+ break;
+ auto *UncountableBR =
+ cast<BranchInst>(UncountableExits.front()->getTerminator());
+ if (!VPlanTransforms::runPass(
+ VPlanTransforms::handleUncountableExitsWithSideEffects, *Plan,
+ UncountableBR))
+ break;
+ }
assert(verifyVPlanIsValid(*Plan) && "VPlan is invalid");
VPlans.push_back(std::move(Plan));
}
@@ -8392,7 +8417,8 @@ VPlanPtr LoopVectorizationPlanner::tryToBuildVPlanWithVPRecipes(
return !CM.requiresScalarEpilogue(VF.isVector());
},
Range);
- VPlanTransforms::handleEarlyExits(*Plan, Legal->hasUncountableEarlyExit());
+ VPlanTransforms::handleEarlyExits(*Plan, Legal->hasUncountableEarlyExit(),
+ Legal->hasUncountableExitWithSideEffects());
VPlanTransforms::addMiddleCheck(*Plan, RequiresScalarEpilogueCheck,
CM.foldTailByMasking());
@@ -8657,8 +8683,10 @@ VPlanPtr LoopVectorizationPlanner::tryToBuildVPlan(VFRange &Range) {
MapVector<PHINode *, RecurrenceDescriptor>(),
SmallPtrSet<const PHINode *, 1>(), SmallPtrSet<PHINode *, 1>(),
/*AllowReordering=*/false);
- VPlanTransforms::handleEarlyExits(*Plan,
- /*HasUncountableExit*/ false);
+ VPlanTransforms::handleEarlyExits(
+ *Plan,
+ /*HasUncountableExit*/ false,
+ /*HasUncountableExitWithSideEffects=*/false);
VPlanTransforms::addMiddleCheck(*Plan, /*RequiresScalarEpilogue*/ true,
/*TailFolded*/ false);
@@ -9767,6 +9795,14 @@ bool LoopVectorizePass::processLoop(Loop *L) {
"UncountableEarlyExitLoopsDisabled", ORE, L);
return false;
}
+ if (LVL.hasUncountableExitWithSideEffects() &&
+ !EnableEarlyExitWithSideEffectVectorization) {
+ reportVectorizationFailure("Auto-vectorization of loops with uncountable "
+ "early exit and side effects is not enabled",
+ "UncountableEarlyExitSideEffectLoopsDisabled",
+ ORE, L);
+ return false;
+ }
SmallVector<BasicBlock *, 8> ExitingBlocks;
L->getExitingBlocks(ExitingBlocks);
// TODO: Support multiple uncountable early exits.
@@ -10034,6 +10070,14 @@ bool LoopVectorizePass::processLoop(Loop *L) {
IC = 1;
}
+ if (InterleaveLoop && LVL.hasUncountableExitWithSideEffects()) {
+ LLVM_DEBUG(dbgs() << "LV: Not interleaving due to EE with side effects.\n");
+ IntDiagMsg = {"EEWithSideEffectsPreventsInterleaving",
+ "Unable to interleave due to early exit with side effects."};
+ InterleaveLoop = false;
+ IC = 1;
+ }
+
// Emit diagnostic messages, if any.
const char *VAPassName = Hints.vectorizeAnalysisPassName();
if (!VectorizeLoop && !InterleaveLoop) {
diff --git a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
index 4846abdab3c55..f3b40707d8156 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp
@@ -867,7 +867,8 @@ void VPlanTransforms::createInLoopReductionRecipes(
}
void VPlanTransforms::handleEarlyExits(VPlan &Plan,
- bool HasUncountableEarlyExit) {
+ bool HasUncountableEarlyExit,
+ bool HasUncountableExitWithSideEffects) {
auto *MiddleVPBB = cast<VPBasicBlock>(
Plan.getScalarHeader()->getSinglePredecessor()->getPredecessors()[0]);
auto *LatchVPBB = cast<VPBasicBlock>(MiddleVPBB->getSinglePredecessor());
@@ -887,7 +888,8 @@ void VPlanTransforms::handleEarlyExits(VPlan &Plan,
assert(!HandledUncountableEarlyExit &&
"can handle exactly one uncountable early exit");
handleUncountableEarlyExit(cast<VPBasicBlock>(Pred), EB, Plan,
- cast<VPBasicBlock>(HeaderVPB), LatchVPBB);
+ cast<VPBasicBlock>(HeaderVPB), LatchVPBB,
+ HasUncountableExitWithSideEffects);
HandledUncountableEarlyExit = true;
} else {
for (VPRecipeBase &R : EB->phis())
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
index c09e0d4e1ad94..d73556ae664db 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp
@@ -3944,13 +3944,11 @@ void VPlanTransforms::convertToConcreteRecipes(VPlan &Plan) {
R->eraseFromParent();
}
-void VPlanTransforms::handleUncountableEarlyExit(VPBasicBlock *EarlyExitingVPBB,
- VPBasicBlock *EarlyExitVPBB,
- VPlan &Plan,
- VPBasicBlock *HeaderVPBB,
- VPBasicBlock *LatchVPBB) {
+void VPlanTransforms::handleUncountableEarlyExit(
+ VPBasicBlock *EarlyExitingVPBB, VPBasicBlock *EarlyExitVPBB, VPlan &Plan,
+ VPBasicBlock *HeaderVPBB, VPBasicBlock *LatchVPBB, bool HasSideEffects) {
auto *MiddleVPBB = cast<VPBasicBlock>(LatchVPBB->getSuccessors()[0]);
- if (!EarlyExitVPBB->getSinglePredecessor() &&
+ if (!HasSideEffects && !EarlyExitVPBB->getSinglePredecessor() &&
EarlyExitVPBB->getPredecessors()[1] == MiddleVPBB) {
assert(EarlyExitVPBB->getNumPredecessors() == 2 &&
EarlyExitVPBB->getPredecessors()[0] == EarlyExitingVPBB &&
@@ -3976,6 +3974,23 @@ void VPlanTransforms::handleUncountableEarlyExit(VPBasicBlock *EarlyExitingVPBB,
// [0] vector.early.exit, [1] middle block, [2] header (continue looping).
VPValue *IsEarlyExitTaken =
Builder.createNaryOp(VPInstruction::AnyOf, {CondToEarlyExit});
+
+ auto *LatchExitingBranch = cast<VPInstruction>(LatchVPBB->getTerminator());
+ assert(LatchExitingBranch->getOpcode() == VPInstruction::BranchOnCount &&
+ "Unexpected terminator");
+ auto *IsLatchExitTaken =
+ Builder.createICmp(CmpInst::ICMP_EQ, LatchExitingBranch->getOperand(0),
+ LatchExitingBranch->getOperand(1));
+ DebugLoc LatchDL = LatchExitingBranch->getDebugLoc();
+ LatchExitingBranch->eraseFromParent();
+
+ if (HasSideEffects) {
+ Builder.setInsertPoint(LatchVPBB);
+ VPInstruction *Combined = Builder.createNaryOp(
+ Instruction::Or, {IsEarlyExitTaken, IsLatchExitTaken}, LatchDL);
+ Builder.createNaryOp(VPInstruction::BranchOnCond, {Combined});
+ return;
+ }
VPBasicBlock *VectorEarlyExitVPBB =
Plan.createVPBasicBlock("vector.early.exit");
VectorEarlyExitVPBB->setParent(EarlyExitVPBB->getParent());
@@ -4013,15 +4028,6 @@ void VPlanTransforms::handleUncountableEarlyExit(VPBasicBlock *EarlyExitingVPBB,
// loop with a multi-conditional branch exiting to vector early exit if the
// early exit has been taken, exiting to middle block if the original
// condition of the vector latch is true, otherwise continuing back to header.
- auto *LatchExitingBranch = cast<VPInstruction>(LatchVPBB->getTerminator());
- assert(LatchExitingBranch->getOpcode() == VPInstruction::BranchOnCount &&
- "Unexpected terminator");
- auto *IsLatchExitTaken =
- Builder.createICmp(CmpInst::ICMP_EQ, LatchExitingBranch->getOperand(0),
- LatchExitingBranch->getOperand(1));
-
- DebugLoc LatchDL = LatchExitingBranch->getDebugLoc();
- LatchExitingBranch->eraseFromParent();
Builder.setInsertPoint(LatchVPBB);
Builder.createNaryOp(VPInstruction::BranchOnTwoConds,
@@ -4031,6 +4037,160 @@ void VPlanTransforms::handleUncountableEarlyExit(VPBasicBlock *EarlyExitingVPBB,
VectorEarlyExitVPBB->setPredecessors({LatchVPBB});
}
+bool VPlanTransforms::handleUncountableExitsWithSideEffects(
+ VPlan &Plan, BranchInst *UncountableBR) {
+ if (Plan.hasScalarVFOnly())
+ return false;
+
+ // We can abandon a vplan entirely if we return false here, so we shouldn't
+ // crash if some earlier assumptions on scalar IR don't hold for the vplan
+ // version of the loop.
+ VPCanonicalIVPHIRecipe *IV = Plan.getVectorLoopRegion()->getCanonicalIV();
+ VPInstruction *IVUpdate = dyn_cast<VPInstruction>(IV->getBackedgeValue());
+ if (!IVUpdate)
+ return false;
+
+ SmallVector<VPRecipeBase *, 2> GEPs;
+ SmallVector<VPRecipeBase *, 8> ConditionRecipes;
+
+ std::optional<VPValue *> Cond =
+ vputils::getRecipesForUncountableExit(Plan, ConditionRecipes, GEPs);
+ if (!Cond)
+ return false;
+
+ // Find load contributing to condition.
+ VPWidenLoadRecipe *CondLoad = nullptr;
+ for (auto *Recipe : ConditionRecipes) {
+ if (auto *L = dyn_cast<VPWidenLoadRecipe>(Recipe)) {
+ // TODO: Support more than one load. Needs legality updates too.
+ assert(CondLoad == nullptr && "Too many condition loads\n");
+ CondLoad = L;
+ }
+ }
+ assert(CondLoad && "Couldn't find load\n");
+
+ // Check GEPs to see if we can link them to the canonical IV.
+ using namespace llvm::VPlanPatternMatch;
+ for (auto *GEP : GEPs)
+ if (!match(GEP,
+ m_GetElementPtr(m_LiveIn(),
+ m_ScalarIVSteps(m_Specific(IV), m_SpecificInt(1),
+ m_Specific(&Plan.getVF())))))
+ return false;
+
+ // Do we need to move? Find the first memory operation that isn't the
+ // load for the early exit condition comparison.
+ VPRecipeBase *FirstMemOp = nullptr;
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_deep(Plan.getVectorLoopRegion()))) {
+ for (VPRecipeBase &R : make_early_inc_range(*VPBB)) {
+ if (auto *MemOp = dyn_cast<VPWidenMemoryRecipe>(&R);
+ MemOp != nullptr && MemOp != CondLoad) {
+ FirstMemOp = &R;
+ break;
+ }
+ }
+
+ if (FirstMemOp)
+ break;
+ }
+
+ // TODO: Assert instead?
+ if (!FirstMemOp)
+ return false;
+
+ // If another memory operation would take place before the comparison to
+ // determine whether to exit early, move the comparison (and supporting
+ // recipes) before it.)
+ VPRecipeBase *CondR = (*Cond)->getDefiningRecipe();
+ VPDominatorTree VPDT(Plan);
+ if (!VPDT.properlyDominates(CondR, FirstMemOp)) {
+ for (auto *Recipe : reverse(ConditionRecipes))
+ Recipe->moveBefore(*FirstMemOp->getParent(), FirstMemOp->getIterator());
+
+ CondR->moveBefore(*FirstMemOp->getParent(), FirstMemOp->getIterator());
+ }
+
+ // Create a mask to represent all lanes that fully execute in the vector loop,
+ // stopping short of any early exit.
+ VPBuilder MaskBuilder(FirstMemOp);
+ VPValue *FirstActive =
+ MaskBuilder.createNaryOp(VPInstruction::FirstActiveLane, *Cond);
+ VPValue *ALMMultiplier =
+ Plan.getConstantInt(Plan.getVectorLoopRegion()->getCanonicalIVType(), 1);
+ VPValue *Zero =
+ Plan.getConstantInt(Plan.getVectorLoopRegion()->getCanonicalIVType(), 0);
+ VPValue *Mask = MaskBuilder.createNaryOp(VPInstruction::ActiveLaneMask,
+ {Zero, FirstActive, ALMMultiplier},
+ nullptr, "uncountable.exit.mask");
+
+ // Convert all other memory operations to use the mask.
+ // TODO: If a (non-condition) load is guaranteed to be dereferenceable, does
+ // it matter if we don't convert it?
+ for (VPBasicBlock *VPBB : VPBlockUtils::blocksOnly<VPBasicBlock>(
+ vp_depth_first_deep(Plan.getVectorLoopRegion()))) {
+ for (VPRecipeBase &R : make_early_inc_range(*VPBB)) {
+ if (auto *MemOp = dyn_cast<VPWidenMemoryRecipe>(&R)) {
+ if (MemOp == CondLoad)
+ continue;
+
+ if (auto *St = dyn_cast<VPWidenStoreRecipe>(MemOp)) {
+ auto *NewStore = new VPWidenStoreRecipe(
+ (cast<StoreInst>(St->getIngredient())), St->getAddr(),
+ St->getStoredValue(), Mask, St->isConsecutive(), St->isReverse(),
+ *St, St->getDebugLoc());
+ NewStore->insertBefore(St);
+ St->eraseFromParent();
+ } else if (auto *Ld = dyn_cast<VPWidenLoadRecipe>(MemOp)) {
+ auto *NewLoad = new VPWidenLoadRecipe(
+ *(cast<LoadInst>(Ld->getUnderlyingValue())), Ld->getOperand(0),
+ Mask, Ld->isConsecutive(), Ld->isReverse(), *Ld,
+ Ld->getDebugLoc());
+ NewLoad->insertBefore(Ld);
+ Ld->replaceAllUsesWith(NewLoad);
+ Ld->eraseFromParent();
+ } else {
+ llvm_unreachable("wrong memop?");
+ }
+ }
+ }
+ }
+
+ // Update middle block branch to compare (IV + however many lanes were active)
+ // against the full trip count, since we may be exiting the vector loop early.
+ // If we didn't take an early exit, we should get the equivalent of VF from
+ // the cttz.elts.
+ VPBasicBlock *MiddleBlock = Plan.getMiddleBlock();
+ VPRecipeBase *OldTerminator = MiddleBlock->getTerminator();
+ VPBuilder MiddleBuilder(OldTerminator);
+ // TODO: Create popcount of mask another way... zext + reduce_add?
+ // For SVE at least we want to be able to fold away the
+ // brkb+cntp+whilelo in the loop to just brkb. We need a cntp
+ // outside the loop instead... maybe copy in CodeGenPrepare?
+ VPTypeAnalysis TypeInfo(Plan);
+ VPValue *ExitIV =
+ MiddleBuilder.createNaryOp(Instruction::Add, {IV, FirstActive});
+ VPValue *FullTC =
+ MiddleBuilder.createICmp(CmpInst::ICMP_EQ, ExitIV, Plan.getTripCount());
+ OldTerminator->setOperand(0, FullTC);
+
+ // Update resume phi in scalar.ph...
+ VPBasicBlock *ScalarPH = Plan.getScalarPreheader();
+ auto Phis = ScalarPH->phis();
+ bool PhiFound = false;
+ for (auto &Phi : Phis) {
+ // TODO: Handle more than one Phi; re-derive from IV.
+ // TODO: Handle reductions, one day...
+ if (PhiFound)
+ return false;
+ PhiFound = true;
+ VPPhi *ContinueIV = cast<VPPhi>(&Phi);
+ ContinueIV->setOperand(0, ExitIV);
+ }
+
+ return true;
+}
+
/// This function tries convert extended in-loop reductions to
/// VPExpressionRecipe and clamp the \p Range if it is beneficial and
/// valid. The created recipe must be decomposed to its constituent
diff --git a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
index e0d09a099647a..cca2eb269b296 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
+++ b/llvm/lib/Transforms/Vectorize/VPlanTransforms.h
@@ -123,8 +123,9 @@ struct VPlanTransforms {
ElementCount MinVF);
/// Update \p Plan to account for all early exits.
- LLVM_ABI_FOR_TEST static void handleEarlyExits(VPlan &Plan,
- bool HasUncountableExit);
+ LLVM_ABI_FOR_TEST static void
+ handleEarlyExits(VPlan &Plan, bool HasUncountableExit,
+ bool HasUncountableExitWithSideEffects);
/// If a check is needed to guard executing the scalar epilogue loop, it will
/// be added to the middle block.
@@ -299,7 +300,13 @@ struct VPlanTransforms {
static void handleUncountableEarlyExit(VPBasicBlock *EarlyExitingVPBB,
VPBasicBlock *EarlyExitVPBB,
...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/178454
More information about the llvm-commits
mailing list