[llvm] [LV][NFC] Query UnrollVectorizedLoop before executing the plan (PR #208500)
Igor Kirillov via llvm-commits
llvm-commits at lists.llvm.org
Thu Jul 9 09:37:07 PDT 2026
https://github.com/igogo-x86 created https://github.com/llvm/llvm-project/pull/208500
LoopVectorize decides whether to attach llvm.loop.unroll.runtime.disable metadata to the vector loop based on a single target-wide bit, UnrollingPreferences::UnrollVectorizedLoop (AMDGPU sets it unconditionally since D149281). To read this bit, it called getUnrollingPreferences on the newly created vector loop and passed the vectorizer's ScalarEvolution, so SCEV was queried on IR in the middle of the transform.
This has a visible side effect: any SCEV query at this point fills the SCEV caches with expressions for the new loop skeleton, and this changes which existing values SCEVExpander reuses later when it expands the minimum-iteration checks for the epilogue. As a result, if a target's getUnrollingPreferences queries SCEV (ARM already does, AArch64 will in llvm/llvm-project#205102), the emitted IR depends on when the hook runs.
UnrollVectorizedLoop is a target constant, so query it earlier, on the original loop - before the plan executes and before the original loop's SCEV is forgotten. At that point the query is a cache hit and does not add any new vector-loop expressions to the caches.
>From f151c32c2537d2d72a6a4517296ee3ec1755238c Mon Sep 17 00:00:00 2001
From: Igor Kirillov <igor.kirillov at arm.com>
Date: Thu, 9 Jul 2026 16:15:33 +0000
Subject: [PATCH] [LV][NFC] Query UnrollVectorizedLoop before executing the
plan
LoopVectorize decides whether to attach llvm.loop.unroll.runtime.disable
metadata to the vector loop based on a single target-wide bit,
UnrollingPreferences::UnrollVectorizedLoop (AMDGPU sets it
unconditionally since D149281). To read this bit, it called
getUnrollingPreferences on the newly created vector loop and passed the
vectorizer's ScalarEvolution, so SCEV was queried on IR in the middle
of the transform.
This has a visible side effect: any SCEV query at this point fills the
SCEV caches with expressions for the new loop skeleton, and this
changes which existing values SCEVExpander reuses later when it
expands the minimum-iteration checks for the epilogue. As a result, if
a target's getUnrollingPreferences queries SCEV (ARM already does,
AArch64 will in llvm/llvm-project#205102), the emitted IR depends on
when the hook runs.
UnrollVectorizedLoop is a target constant, so query it earlier, on the
original loop - before the plan executes and before the original
loop's SCEV is forgotten. At that point the query is a cache hit and
does not add any new vector-loop expressions to the caches.
---
.../Transforms/Vectorize/LoopVectorizationPlanner.h | 5 +++--
llvm/lib/Transforms/Vectorize/LoopVectorize.cpp | 10 +++++++++-
llvm/lib/Transforms/Vectorize/VPlan.cpp | 6 ++----
3 files changed, 14 insertions(+), 7 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
index c5524d01fa76e..363824ec9e0c6 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorizationPlanner.h
@@ -981,13 +981,14 @@ class LoopVectorizationPlanner {
/// the average trip count and invocation weight of the original loop (\p
/// OrigAverageTripCount and \p OrigLoopInvocationWeight respectively). They
/// cannot be retrieved after the plan has been executed, as the original loop
- /// may have been removed.
+ /// may have been removed. \p UnrollVectorizedLoop, whether the target wants
+ /// the vector loop left eligible for runtime unrolling.
void updateLoopMetadataAndProfileInfo(
Loop *VectorLoop, VPBasicBlock *HeaderVPBB, const VPlan &Plan,
bool VectorizingEpilogue, MDNode *OrigLoopID,
std::optional<unsigned> OrigAverageTripCount,
unsigned OrigLoopInvocationWeight, unsigned EstimatedVFxUF,
- bool DisableRuntimeUnroll);
+ bool DisableRuntimeUnroll, bool UnrollVectorizedLoop);
private:
/// Build an initial VPlan, with HCFG wrapping the original scalar loop and
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 19b77e40a4103..ca424b0233dfa 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -6038,6 +6038,14 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
SE.forgetLcssaPhiWithNewPredecessor(OrigLoop,
&cast<VPIRPhi>(PhiR).getIRPhi());
}
+
+ // Query whether the target wants loops it vectorizes to remain eligible for
+ // runtime unrolling. Do this here, on the original loop and before its SCEV
+ // is forgotten below.
+ TargetTransformInfo::UnrollingPreferences UP;
+ TTI.getUnrollingPreferences(OrigLoop, SE, UP, ORE);
+ bool UnrollVectorizedLoop = UP.UnrollVectorizedLoop;
+
// Forget the original loop and block dispositions.
SE.forgetLoop(OrigLoop);
SE.forgetBlockAndLoopDispositions();
@@ -6076,7 +6084,7 @@ DenseMap<const SCEV *, Value *> LoopVectorizationPlanner::executePlan(
EpilogueVecKind == EpilogueVectorizationKind::Epilogue, LID,
OrigAverageTripCount, OrigLoopInvocationWeight,
estimateElementCount(BestVF * BestUF, Config.getVScaleForTuning()),
- DisableRuntimeUnroll);
+ DisableRuntimeUnroll, UnrollVectorizedLoop);
// 3. Fix the vectorized code: take care of header phi's, live-outs,
// predication, updating analyses.
diff --git a/llvm/lib/Transforms/Vectorize/VPlan.cpp b/llvm/lib/Transforms/Vectorize/VPlan.cpp
index 2088f91657289..94c229bdfa59d 100644
--- a/llvm/lib/Transforms/Vectorize/VPlan.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlan.cpp
@@ -1742,7 +1742,7 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
bool VectorizingEpilogue, MDNode *OrigLoopID,
std::optional<unsigned> OrigAverageTripCount,
unsigned OrigLoopInvocationWeight, unsigned EstimatedVFxUF,
- bool DisableRuntimeUnroll) {
+ bool DisableRuntimeUnroll, bool UnrollVectorizedLoop) {
// Update the metadata of the scalar loop. Skip the update when vectorizing
// the epilogue loop to ensure it is updated only once. Also skip the update
// when the scalar loop became unreachable.
@@ -1791,9 +1791,7 @@ void LoopVectorizationPlanner::updateLoopMetadataAndProfileInfo(
// emit when remarks are enabled.
if (ORE->enabled())
VectorLoop->addIntLoopAttribute("llvm.loop.vectorize.body", 1);
- TargetTransformInfo::UnrollingPreferences UP;
- TTI.getUnrollingPreferences(VectorLoop, *PSE.getSE(), UP, ORE);
- if (!UP.UnrollVectorizedLoop || VectorizingEpilogue)
+ if (!UnrollVectorizedLoop || VectorizingEpilogue)
addRuntimeUnrollDisableMetaData(VectorLoop);
// Set/update profile weights for the vector and remainder loops as original
More information about the llvm-commits
mailing list