[llvm] [LoopVectorize] Introduce check-first vectorization for multiple early-exit loops (PR #210492)
via llvm-commits
llvm-commits at lists.llvm.org
Fri Jul 17 23:59:08 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-vectorizers
Author: Arjun H Kumar (arjun-harikumar-amd)
<details>
<summary>Changes</summary>
### Check-First Early-Exit Loop Vectorization
We introduce check-first strategy for vectorizing loops with multiple early exits. Instead of masking the loop body, the check-first approach evaluates all exit conditions first for a vector chunk and only runs the loop body if no exit fires. This generalizes early-exit vectorization to a much broader set of loops.
Implemented as a VPlan transformation in the loop vectorizer. It restructures the vector loop into three regions:
1. **Check blocks**: One check block per early exit, each holding the minimal condition slice to evaluate that exit. If any exit fires, control transfers to the exit-handling path.
2. **Vector body**: Runs only when no exit fires.
3. **Exit handling**: For the exiting chunk, lanes before the exit still need to execute. We have two strategies:
- **Scalar replay** (default): fall back to the scalar loop, replaying from the start of the current chunk.
- **Masked replay** (experimental): clone the body's stores with lane masks derived from the first active exit lane, avoiding the scalar loop.
### Flags
```
-mllvm -enable-check-first-early-exit-vectorization
-mllvm -enable-check-first-masked-replay
```
RFC to be posted soon.
Co-authored by @<!-- -->nema-ashutosh
---
Patch is 97.87 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/210492.diff
14 Files Affected:
- (modified) llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h (+22)
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp (+297-3)
- (modified) llvm/lib/Transforms/Vectorize/LoopVectorize.cpp (+93-18)
- (modified) llvm/lib/Transforms/Vectorize/VPlan.cpp (+28)
- (modified) llvm/lib/Transforms/Vectorize/VPlan.h (+83-8)
- (modified) llvm/lib/Transforms/Vectorize/VPlanConstruction.cpp (+17-2)
- (modified) llvm/lib/Transforms/Vectorize/VPlanPredicator.cpp (+2)
- (modified) llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp (+12-1)
- (modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.cpp (+931-4)
- (modified) llvm/lib/Transforms/Vectorize/VPlanTransforms.h (+10)
- (modified) llvm/lib/Transforms/Vectorize/VPlanVerifier.cpp (+6)
- (modified) llvm/test/Transforms/LoopVectorize/VPlan/vplan-print-before-after-all.ll (+1)
- (added) llvm/test/Transforms/LoopVectorize/check-first-multi-exit-cascade.ll (+133)
- (added) llvm/test/Transforms/LoopVectorize/check-first-nested-exit.ll (+205)
``````````diff
diff --git a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
index 3e8db73fd79d2..de7eb4466ef6b 100644
--- a/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
+++ b/llvm/include/llvm/Transforms/Vectorize/LoopVectorizationLegality.h
@@ -440,6 +440,20 @@ class LoopVectorizationLegality {
return getUncountableExitTrait() == UncountableExitTrait::ReadWrite;
}
+ /// Returns true if every widened exit condition load is
+ /// dereferenceable for the complete trip count.
+ bool exitLoadsAreDereferenceable() const {
+ return AllExitLoadsDereferenceable;
+ }
+
+ /// Returns true if this early exit loop would use the check first strategy if
+ /// enabled.
+ bool wouldUseCheckFirstStyle() const {
+ if (hasUncountableExitWithSideEffects())
+ return true;
+ return hasUncountableEarlyExit() && !AllExitLoadsDereferenceable;
+ }
+
/// Return true if there is store-load forwarding dependencies.
bool isSafeForAnyStoreLoadForwardDistances() const {
return LAI->getDepChecker().isSafeForAnyStoreLoadForwardDistances();
@@ -635,6 +649,10 @@ class LoopVectorizationLegality {
/// for it.
bool canUncountableExitConditionLoadBeMoved(BasicBlock *ExitingBlock);
+ /// Returns true if the exit conditions can be safely speculated.
+ bool canCheckFirstSpeculateExitConditions(
+ ArrayRef<BasicBlock *> ExitingBlocks);
+
/// Return true if all of the instructions in the block can be speculatively
/// executed, and record the loads/stores that require masking.
/// \p SafePtrs is a list of addresses that are known to be legal and we know
@@ -752,6 +770,10 @@ class LoopVectorizationLegality {
/// Records whether we have an uncountable early exit in a loop that's
/// either read-only or read-write.
UncountableExitTrait UncountableExitType = UncountableExitTrait::None;
+
+ /// Records whether every widened exit condition load is
+ /// dereferenceable for the complete trip count.
+ bool AllExitLoadsDereferenceable = true;
};
} // namespace llvm
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
index 8d875b2b6e492..c94de8d9951b3 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationLegality.cpp
@@ -79,6 +79,14 @@ static cl::opt<bool> EnableHistogramVectorization(
"enable-histogram-loop-vectorization", cl::init(false), cl::Hidden,
cl::desc("Enables autovectorization of some loops containing histograms"));
+static cl::opt<unsigned> MaxUncountableEarlyExits(
+ "max-uncountable-early-exits", cl::init(4), cl::Hidden,
+ cl::desc("Maximum number of uncountable early exits a loop may have to be "
+ "eligible for multi-exit check-first vectorization."));
+
+extern cl::opt<bool> EnableCheckFirstVectorization;
+extern cl::opt<bool> EnableEarlyExitVectorizationWithSideEffects;
+
/// Maximum vectorization interleave count.
static const unsigned MaxInterleaveFactor = 16;
@@ -1616,6 +1624,98 @@ bool LoopVectorizationLegality::canVectorizeLoopNestCFG(
return Result;
}
+static unsigned countInLoopPredecessors(const BasicBlock *BB, const Loop *L) {
+ unsigned Count = 0;
+ for (const BasicBlock *Pred : predecessors(BB))
+ if (L->contains(Pred))
+ ++Count;
+ return Count;
+}
+
+/// Walks up the dominator chain from \p Exiting to the loop header, collecting
+/// into \p GuardConds the branch conditions that must hold for control to reach
+/// \p ExitingBB. Returns false if the control flow has a shape that cannot be
+/// represented as such a guard.
+///
+/// At each conditional dominator, one successor leads toward \p ExitingBB and the
+/// other bypasses it.
+/// Records a guard when the bypass leaves the guarded region cleanly:
+/// it targets \p Latch, or rejoins the chain after an if-without-else.
+static bool collectExitGuards(BasicBlock *Exiting, Loop *L, BasicBlock *Latch,
+ const DominatorTree &DT,
+ SmallVectorImpl<Value *> &GuardConds,
+ bool AllowRejoin) {
+ BasicBlock *Header = L->getHeader();
+ BasicBlock *Cur = Exiting;
+ while (Cur != Header) {
+ DomTreeNode *Node = DT.getNode(Cur);
+ if (!Node || !Node->getIDom())
+ return false;
+ BasicBlock *IDom = Node->getIDom()->getBlock();
+ if (!L->contains(IDom))
+ return false;
+ if (auto *Br = dyn_cast<CondBrInst>(IDom->getTerminator())) {
+ BasicBlock *S0 = Br->getSuccessor(0), *S1 = Br->getSuccessor(1);
+ bool S0Dom = DT.dominates(S0, Cur), S1Dom = DT.dominates(S1, Cur);
+ if (S0Dom != S1Dom) {
+ BasicBlock *Interior = S0Dom ? S0 : S1;
+ BasicBlock *Bypass = S0Dom ? S1 : S0;
+ if (L->contains(Bypass)) {
+ if (Bypass == Latch) {
+ GuardConds.push_back(Br->getCondition());
+ } else if (!AllowRejoin) {
+ return false;
+ } else if (countInLoopPredecessors(Interior, L) == 1 &&
+ countInLoopPredecessors(Bypass, L) > 1) {
+ GuardConds.push_back(Br->getCondition());
+ } else if (countInLoopPredecessors(Interior, L) > 1) {
+ } else {
+ return false;
+ }
+ }
+ }
+ }
+ Cur = IDom;
+ }
+ return true;
+}
+
+/// Collects the loads feeding the exit conditions of early-exits.
+/// condition, which check-first widens speculatively.
+static void collectExitConditionSliceLoads(ArrayRef<BasicBlock *> ExitingBlocks,
+ Loop *L, BasicBlock *Latch,
+ const DominatorTree &DT,
+ SmallVectorImpl<LoadInst *> &Out) {
+ SmallPtrSet<Value *, 16> Visited;
+ SmallVector<Value *, 16> Worklist;
+ for (BasicBlock *BB : ExitingBlocks) {
+ auto *Br = dyn_cast<CondBrInst>(BB->getTerminator());
+ assert(Br && "exiting block must terminate with a conditional branch");
+ Worklist.push_back(Br->getCondition());
+ collectExitGuards(BB, L, Latch, DT, Worklist, /*AllowRejoin=*/true);
+ }
+
+ for (BasicBlock *BB : L->blocks())
+ for (Instruction &I : *BB)
+ if (isa<StoreInst>(&I) && !DT.dominates(BB, Latch))
+ collectExitGuards(BB, L, Latch, DT, Worklist, /*AllowRejoin=*/false);
+
+ while (!Worklist.empty()) {
+ Value *V = Worklist.pop_back_val();
+ if (!Visited.insert(V).second)
+ continue;
+ auto *I = dyn_cast<Instruction>(V);
+ if (!I || !L->contains(I) || isa<PHINode>(I))
+ continue;
+ if (auto *LI = dyn_cast<LoadInst>(I)) {
+ Out.push_back(LI);
+ continue;
+ }
+ for (Value *Op : I->operands())
+ Worklist.push_back(Op);
+ }
+}
+
bool LoopVectorizationLegality::isVectorizableEarlyExitLoop() {
BasicBlock *LatchBB = TheLoop->getLoopLatch();
if (!LatchBB) {
@@ -1733,10 +1833,15 @@ bool LoopVectorizationLegality::isVectorizableEarlyExitLoop() {
return false;
}
} else {
- // Check all uncountable exiting blocks for movable loads.
- for (BasicBlock *ExitingBB : UncountableExitingBlocks) {
- if (!canUncountableExitConditionLoadBeMoved(ExitingBB))
+ if (EnableCheckFirstVectorization &&
+ !EnableEarlyExitVectorizationWithSideEffects) {
+ if (!canCheckFirstSpeculateExitConditions(UncountableExitingBlocks))
return false;
+ } else {
+ for (BasicBlock *ExitingBB : UncountableExitingBlocks) {
+ if (!canUncountableExitConditionLoadBeMoved(ExitingBB))
+ return false;
+ }
}
}
@@ -1754,6 +1859,57 @@ bool LoopVectorizationLegality::isVectorizableEarlyExitLoop() {
}
}
+ // Safe only if every widened condition slice load is dereferenceable.
+ if (HasSideEffects) {
+ SmallVector<LoadInst *, 4> SpeculatedCondLoads;
+ collectExitConditionSliceLoads(UncountableExitingBlocks, TheLoop,
+ TheLoop->getLoopLatch(), *DT,
+ SpeculatedCondLoads);
+
+ bool AllDeref = true;
+ for (LoadInst *LI : SpeculatedCondLoads) {
+ if (!isDereferenceableAndAlignedInLoop(LI, TheLoop, *PSE.getSE(), *DT,
+ AC)) {
+ AllDeref = false;
+ break;
+ }
+ }
+
+ AllExitLoadsDereferenceable = AllDeref;
+
+ LLVM_DEBUG({
+ dbgs() << "LV: check-first early-exit memory-safety strategy: ";
+ if (AllDeref)
+ dbgs() << "all speculated condition-slice loads provably "
+ "dereferenceable. \n";
+ else
+ dbgs() << "Condition-slice loads not provably dereferenceable. \n";
+ });
+ }
+
+ bool WillUseCheckFirst =
+ HasSideEffects && !EnableEarlyExitVectorizationWithSideEffects;
+ if (WillUseCheckFirst) {
+ const InductionDescriptor *IndDesc = nullptr;
+ if (Inductions.size() == 1) {
+ IndDesc = &Inductions.begin()->second;
+ } else if (PHINode *PrimaryIV = getPrimaryInduction()) {
+ auto It = Inductions.find(PrimaryIV);
+ if (It != Inductions.end())
+ IndDesc = &It->second;
+ }
+ if (!IndDesc ||
+ (IndDesc->getKind() != InductionDescriptor::IK_IntInduction &&
+ IndDesc->getKind() != InductionDescriptor::IK_PtrInduction) ||
+ !IndDesc->getConstIntStepValue()) {
+ reportVectorizationFailure(
+ "Check-first early-exit vectorization requires a single integer or "
+ "pointer induction with a constant step",
+ "UnsupportedCheckFirstInduction", ORE, TheLoop);
+ return false;
+ }
+ }
+
[[maybe_unused]] const SCEV *SymbolicMaxBTC =
PSE.getSymbolicMaxBackedgeTakenCount();
// Since we have an exact exit count for the latch and the early exit
@@ -1854,6 +2010,144 @@ bool LoopVectorizationLegality::canUncountableExitConditionLoadBeMoved(
return true;
}
+bool LoopVectorizationLegality::canCheckFirstSpeculateExitConditions(
+ ArrayRef<BasicBlock *> ExitingBlocks) {
+ if (ExitingBlocks.size() > MaxUncountableEarlyExits) {
+ reportVectorizationFailure(
+ "Too many uncountable early exits for check-first vectorization",
+ "TooManyEarlyExitsForCheckFirst", ORE, TheLoop);
+ return false;
+ }
+
+ BasicBlock *Latch = TheLoop->getLoopLatch();
+ if (!Latch) {
+ reportVectorizationFailure("Check-first early-exit loop has no latch",
+ "NoLatchCheckFirstExit", ORE, TheLoop);
+ return false;
+ }
+
+ SmallVector<Value *, 8> GuardConds;
+ for (BasicBlock *BB : ExitingBlocks) {
+ if (!collectExitGuards(BB, TheLoop, Latch, *DT, GuardConds,
+ /*AllowRejoin=*/true)) {
+ reportVectorizationFailure(
+ "Check-first early-exit vectorization does not support this guarded "
+ "(conditionally-executed) early-exit control-flow shape",
+ "UnsupportedGuardedCheckFirstExit", ORE, TheLoop);
+ return false;
+ }
+ }
+
+ for (BasicBlock *BB : TheLoop->blocks())
+ for (Instruction &I : *BB)
+ if (isa<StoreInst>(&I) && !DT->dominates(BB, Latch)) {
+ SmallVector<Value *, 4> StoreGuards;
+ if (!collectExitGuards(BB, TheLoop, Latch, *DT, StoreGuards,
+ /*AllowRejoin=*/false)) {
+ reportVectorizationFailure(
+ "Check-first early-exit vectorization does not support this "
+ "conditionally-executed (guarded) store control-flow shape",
+ "GuardedCheckFirstStore", ORE, TheLoop);
+ return false;
+ }
+ }
+
+ SmallPtrSet<LoadInst *, 8> CondLoads;
+ SmallVector<Value *, 16> Worklist;
+ SmallPtrSet<Value *, 16> Visited;
+ for (BasicBlock *BB : ExitingBlocks) {
+ auto *Br = dyn_cast<CondBrInst>(BB->getTerminator());
+ if (!Br) {
+ reportVectorizationFailure(
+ "Exiting block does not terminate with a conditional branch",
+ "UnsupportedCheckFirstExitTerminator", ORE, TheLoop);
+ return false;
+ }
+ Worklist.push_back(Br->getCondition());
+ }
+ append_range(Worklist, GuardConds);
+
+ while (!Worklist.empty()) {
+ Value *V = Worklist.pop_back_val();
+ if (!Visited.insert(V).second)
+ continue;
+ if (TheLoop->isLoopInvariant(V))
+ continue;
+ auto *I = dyn_cast<Instruction>(V);
+ if (!I || !TheLoop->contains(I)) {
+ reportVectorizationFailure(
+ "Early exit condition depends on a value that cannot be "
+ "speculatively evaluated for check-first vectorization",
+ "UnsupportedCheckFirstExitCondition", ORE, TheLoop);
+ return false;
+ }
+ if (auto *LI = dyn_cast<LoadInst>(I)) {
+ const auto *AR = dyn_cast<SCEVAddRecExpr>(
+ PSE.getSE()->getSCEV(LI->getPointerOperand()));
+ if (!LI->isSimple() || !AR || AR->getLoop() != TheLoop ||
+ !AR->isAffine()) {
+ reportVectorizationFailure(
+ "Early exit condition depends on a load that is not a simple "
+ "affine (unit-stride) access",
+ "CheckFirstExitLoadInvariantAddress", ORE, TheLoop);
+ return false;
+ }
+ CondLoads.insert(LI);
+ continue;
+ }
+ if (isa<PHINode>(I)) {
+ if (I->getParent() != TheLoop->getHeader()) {
+ reportVectorizationFailure(
+ "Early exit condition depends on a non-header PHI",
+ "UnsupportedCheckFirstExitCondition", ORE, TheLoop);
+ return false;
+ }
+ continue;
+ }
+ if (I->mayReadOrWriteMemory() || !isSafeToSpeculativelyExecute(I)) {
+ reportVectorizationFailure(
+ "Early exit condition contains an operation that cannot be "
+ "speculatively executed",
+ "UnsupportedCheckFirstExitCondition", ORE, TheLoop);
+ return false;
+ }
+ for (Value *Op : I->operands())
+ Worklist.push_back(Op);
+ }
+
+ SmallPtrSet<const Instruction *, 4> CondLoadSet(CondLoads.begin(),
+ CondLoads.end());
+ ConditionallyExecutedOps.clear();
+ for (auto *BB : TheLoop->blocks()) {
+ for (auto &I : *BB) {
+ if (CondLoadSet.contains(&I) || !I.mayReadOrWriteMemory())
+ continue;
+ ConditionallyExecutedOps.insert(&I);
+ if (isa<LoadInst>(&I))
+ continue;
+ auto *SI = dyn_cast<StoreInst>(&I);
+ if (!SI) {
+ reportVectorizationFailure(
+ "Unsupported memory operation in check-first early-exit loop",
+ "UnsupportedCheckFirstMemOp", ORE, TheLoop);
+ return false;
+ }
+ for (LoadInst *CL : CondLoads) {
+ if (AA->alias(CL->getPointerOperand(), SI->getPointerOperand()) !=
+ AliasResult::NoAlias) {
+ reportVectorizationFailure(
+ "Cannot determine whether an early-exit condition load aliases "
+ "a store (deferred stores must not be observed out of order)",
+ "CheckFirstExitLoadAliasesStore", ORE, TheLoop);
+ return false;
+ }
+ }
+ }
+ }
+
+ return true;
+}
+
bool LoopVectorizationLegality::canVectorize(bool UseVPlanNativePath) {
// Store the result and return it at the end instead of exiting early, in case
// allowExtraAnalysis is used to report multiple reasons for not vectorizing.
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index f1ca4061bfd9e..e7377ff7cea2b 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -411,12 +411,34 @@ static cl::opt<bool> EnableEarlyExitVectorization(
cl::desc(
"Enable vectorization of early exit loops with uncountable exits."));
-static cl::opt<bool> EnableEarlyExitVectorizationWithSideEffects(
+cl::opt<bool> EnableEarlyExitVectorizationWithSideEffects(
"enable-early-exit-vectorization-with-side-effects", cl::init(false),
cl::Hidden,
cl::desc("Enable vectorization of early exit loops with uncountable exits "
"and side effects"));
+cl::opt<bool> EnableCheckFirstVectorization(
+ "enable-check-first-early-exit-vectorization", cl::init(false), cl::Hidden,
+ cl::desc("Enable check-first vectorization of early exit loops with "
+ "multiple exits."));
+
+cl::opt<bool> EnableCheckFirstMaskedReplay(
+ "enable-check-first-masked-replay", cl::init(false), cl::Hidden,
+ cl::desc("Replace scalar replay in check-first early exit vectorization "
+ "with a masked vector replay."));
+
+/// Returns true if loop uses check first with scalar replay of the
+/// failing chunk.
+static bool usesCheckFirstReplay(const LoopVectorizationLegality *Legal) {
+ if (!Legal->hasUncountableEarlyExit())
+ return false;
+ if (EnableCheckFirstMaskedReplay)
+ return false;
+ if (Legal->hasUncountableExitWithSideEffects())
+ return !EnableEarlyExitVectorizationWithSideEffects;
+ return EnableCheckFirstVectorization && Legal->wouldUseCheckFirstStyle();
+}
+
// Likelyhood of bypassing the vectorized loop because there are zero trips left
// after prolog. See `emitIterationCountCheck`.
static constexpr uint32_t MinItersBypassWeights[] = {1, 127};
@@ -3668,6 +3690,11 @@ LoopVectorizationPlanner::selectInterleaveCount(VPlan &Plan, ElementCount VF,
if (Plan.hasEarlyExit())
return 1;
+ // Interleaving would break check-first scalar-replay resume wiring.
+ // So forcing IC=1.
+ if (usesCheckFirstReplay(Legal))
+ return 1;
+
const bool HasReductions =
any_of(Plan.getVectorLoopRegion()->getEntryBasicBlock()->phis(),
IsaPred<VPReductionPHIRecipe>);
@@ -5496,6 +5523,10 @@ void LoopVectorizationPlanner::plan(ElementCount UserVF, unsigned UserIC) {
if (!MaxFactors) // Cases that should not to be vectorized nor interleaved.
return;
+ // Disable scalable vectorization for check-first early-exit loops for now.
+ if (usesCheckFirstReplay(Legal))
+ MaxFactors.ScalableVF = ElementCount::getScalable(0);
+
Config.collectInLoopReductions();
// Cases that may be vectorized may be optimized by unit stride predicates.
// TODO: Currently unit stride predicates are added unconditionally, even if
@@ -5968,6 +5999,11 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
// Regions are dissolved after optimizing for VF and UF, which completely
// removes unneeded loop regions first.
RUN_VPLAN_PASS(VPlanTransforms::dissolveLoopRegions, BestVPlan);
+ // Scalar replay routes check.exit to the scalar preheader after region
+ // dissolution.
+ VPlanTransforms::wireCheckFirstExitToScalar(BestVPlan);
+ // Masked replay routes check.exit to the real early exit block instead.
+ VPlanTransforms::wireCheckFirstMaskedReplayToExit(BestVPlan);
// Expand BranchOnTwoConds after dissolution, when latch has direct access to
// its successors.
RUN_VPLAN_PASS(VPlanTransforms::expandBranchOnTwoConds, BestVPlan);
@@ -6178,8 +6214,6 @@ VPRecipeBase *VPRecipeBuilder::tryToWidenMemory(VPInstruction *VPI,
if (!LoopVectorizationPlanner::getDecisionAndClampRange(WillWiden, Range))
return nullptr;
- // If a mask is not required, drop it - use unmasked version for safe loads.
- // TODO: Determine if mask is needed in VPlan.
VPValue *Mask = CM.isMaskRequired(I) ? VPI->getMask() : nullptr;
// Determine if the pointer operand of the access is either consecutive or
@@ -6580,10 +6614,21 @@ VPlanPtr LoopVectorizationPlanner::tryToBuildVPlan1() {
// the presence of an uncountable exit and the presence of stores in
// the loop inside handleEarlyExits itself.
UncountableExitStyle EEStyle = UncountableExitStyle::NoUncountableExit;
- if (Legal->hasUncountableEarlyExit())
- EEStyle = Legal->hasUncountableExitWithSideEffects()
- ? UncountableExitStyle::MaskedHandleExitInScalarLoop
- : UncountableExitStyle::ReadOnly;
+ if (Legal->hasUncountableEarlyExit()) {
+ if (Legal->hasUncountableExitWithSideEffects()) {
+ if (EnableEarlyExitVectorizationWithSideEffects)
+ EEStyle = UncountableExitStyle::MaskedHandleExitInScalarLoop;
+ else
+ EEStyle = UncountableExitStyle::CheckFirst;
+ } else {
+ EEStyle = UncountableExitStyle::ReadOnly;
+ }
+ }
+
+ assert((EEStyle != UncountableE...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/210492
More information about the llvm-commits
mailing list