[llvm] [SLP] Add store-to-load forwarding conflict cost for widened store chains (PR #199606)
Milin Bhade via llvm-commits
llvm-commits at lists.llvm.org
Mon Sep 28 23:10:10 PDT 2026
https://github.com/mbhade-amd updated https://github.com/llvm/llvm-project/pull/199606
>From b612a00c90ccd0c1609e42224c8aa16c39d1b174 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Fri, 31 Jul 2026 15:10:14 +0530
Subject: [PATCH 01/21] [SLP] Model store-to-load forwarding hazards as a cost
penalty
Widening a consecutive store chain can break store-to-load forwarding
(STLF) for a nearby loop-carried load whose backward distance is
misaligned to the vector store width and short enough that the store is
still in the store buffer, causing a pipeline stall.
Add a target cost hook, getStoreLoadForwardingConflictCost, backed by a
new MCSchedule StoreLoadForwardingConflict penalty (defaulting to 10
cycles, set for znver4). When such a conflict is detected, charge this
penalty to the store entry in the SLP cost model instead of rejecting
the tree outright, so a chain that remains profitable after paying it
can still vectorize.
Conflicting loads are enumerated from the SLP tree via a def-use walk
seeded from the tree scalars and following operands (staying inside the
loop, reaching loads below gather/splat leaves), rather than scanning
the whole loop or LAA's dependence list. The conflict predicate is
shared with LoopAccessAnalysis
(MemoryDepChecker::isStoreLoadForwardingConflict).
---
.../llvm/Analysis/LoopAccessAnalysis.h | 19 +
.../llvm/Analysis/TargetTransformInfo.h | 9 +
.../llvm/Analysis/TargetTransformInfoImpl.h | 9 +
llvm/include/llvm/CodeGen/BasicTTIImpl.h | 7 +
llvm/include/llvm/MC/MCSchedule.h | 11 +
llvm/include/llvm/Target/TargetSchedule.td | 5 +
.../llvm/Transforms/Vectorize/SLPVectorizer.h | 1 +
llvm/lib/Analysis/LoopAccessAnalysis.cpp | 11 +-
llvm/lib/Analysis/TargetTransformInfo.cpp | 14 +
llvm/lib/MC/MCSchedule.cpp | 1 +
llvm/lib/Target/X86/X86ScheduleZnver4.td | 4 +
.../Transforms/Vectorize/SLPVectorizer.cpp | 131 ++++++
.../X86/store-load-forward-conflict.ll | 401 ++++++++++--------
.../X86/store-load-forward-cost.ll | 105 +++++
.../X86/store-load-forward-profitable.ll | 145 +++++++
llvm/utils/TableGen/SubtargetEmitter.cpp | 1 +
16 files changed, 683 insertions(+), 191 deletions(-)
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
diff --git a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
index 392321448c8959..dbd0248f2e58f2 100644
--- a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
+++ b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
@@ -235,6 +235,25 @@ class MemoryDepChecker {
std::numeric_limits<uint64_t>::max();
}
+ /// Returns true if a memory dependence at byte distance \p Distance between
+ /// a store and load (both with element size \p TypeByteSize bytes) would
+ /// prevent store-to-load forwarding when the store is widened to
+ /// \p VectorStoreSize bytes.
+ ///
+ /// The predicate fires when (a) the load is misaligned w.r.t. the widened
+ /// store window (\c Distance is not a multiple of \p VectorStoreSize), and
+ /// (b) the conflicting store is still likely to be in the store buffer
+ /// (\c Distance / VectorStoreSize is below 8 * TypeByteSize iterations).
+ /// Both couldPreventStoreLoadForward and SLPVectorizer use this as their
+ /// core STLF cost-model gate.
+ static bool isStoreLoadForwardingConflict(uint64_t Distance,
+ uint64_t VectorStoreSize,
+ uint64_t TypeByteSize) {
+ const uint64_t NumItersForStoreLoadThroughMemory = 8 * TypeByteSize;
+ return VectorStoreSize != 0 && Distance % VectorStoreSize != 0 &&
+ Distance / VectorStoreSize < NumItersForStoreLoadThroughMemory;
+ }
+
/// Return safe power-of-2 number of elements, which do not prevent store-load
/// forwarding, multiplied by the size of the elements in bits.
uint64_t getStoreLoadForwardSafeDistanceInBits() const {
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index 107ae4dba50751..647ae38de8b727 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1415,6 +1415,15 @@ class TargetTransformInfo {
/// \return The size of a cache line in bytes.
LLVM_ABI unsigned getCacheLineSize() const;
+ /// \return The penalty for a store-to-load forwarding (STLF) hazard when a
+ /// chain of stores is widened into the vector store \p VecTy inside a loop
+ /// and a nearby loop-carried load straddles that widened store, so hardware
+ /// store-to-load forwarding fails and the load stalls until the store drains
+ /// to L1. Targets return the modeled per-occurrence penalty, or 0 if they do
+ /// not model this hazard.
+ LLVM_ABI InstructionCost getStoreLoadForwardingConflictCost(
+ Type *VecTy, TargetCostKind CostKind) const;
+
/// The possible cache levels
enum class CacheLevel {
L1D, // The L1 data cache
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 25574a2d2c0e5b..2d402429a8b8a6 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -23,6 +23,7 @@
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Operator.h"
#include "llvm/IR/PatternMatch.h"
+#include "llvm/MC/MCSchedule.h"
#include <optional>
#include <utility>
@@ -680,6 +681,14 @@ class LLVM_ABI TargetTransformInfoImplBase {
}
virtual unsigned getCacheLineSize() const { return 0; }
+
+ virtual InstructionCost getStoreLoadForwardingConflictCost(
+ Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
+ // No subtarget scheduling model is available here, so fall back to the
+ // conservative default STLF stall penalty (targets with a real model go
+ // through BasicTTIImpl and use their own value instead).
+ return InstructionCost(MCSchedModel::DefaultStoreLoadForwardingPenalty);
+ }
virtual std::optional<unsigned>
getCacheSize(TargetTransformInfo::CacheLevel Level) const {
switch (Level) {
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index cc93f90ff5d8fe..eb0d09f634a444 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -886,6 +886,13 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
return getST()->getCacheLineSize();
}
+ InstructionCost getStoreLoadForwardingConflictCost(
+ Type *VecTy, TTI::TargetCostKind CostKind) const override {
+ // The per-occurrence STLF stall penalty is a microarchitectural cycle
+ // count sourced from the subtarget's scheduling model (0 = not modeled).
+ return InstructionCost(getST()->getSchedModel().StoreLoadForwardingPenalty);
+ }
+
unsigned getPrefetchDistance() const override {
return getST()->getPrefetchDistance();
}
diff --git a/llvm/include/llvm/MC/MCSchedule.h b/llvm/include/llvm/MC/MCSchedule.h
index dd65284671d71f..a065123594bb0a 100644
--- a/llvm/include/llvm/MC/MCSchedule.h
+++ b/llvm/include/llvm/MC/MCSchedule.h
@@ -326,6 +326,17 @@ struct MCSchedModel {
unsigned MispredictPenalty;
static constexpr unsigned DefaultMispredictPenalty = 10;
+ // StoreLoadForwardingPenalty is the typical number of extra cycles a load
+ // stalls when hardware store-to-load forwarding fails (the load's bytes are
+ // not fully contained in a single, recent store, so it must wait for the
+ // store to drain to cache). Consumed by the SLP vectorizer when a widened
+ // store would break forwarding for a nearby loop-carried load.
+ // The failed-forwarding stall is roughly 10-15 cycles on current cores, so
+ // the default uses a conservative 10 cycles for targets that do not override
+ // it; individual targets may set a more precise value (e.g. Zen4 uses 13).
+ unsigned StoreLoadForwardingPenalty;
+ static const unsigned DefaultStoreLoadForwardingPenalty = 10;
+
bool PostRAScheduler; // default value is false
bool CompleteModel;
diff --git a/llvm/include/llvm/Target/TargetSchedule.td b/llvm/include/llvm/Target/TargetSchedule.td
index c68323389b827a..5f23afb2131d72 100644
--- a/llvm/include/llvm/Target/TargetSchedule.td
+++ b/llvm/include/llvm/Target/TargetSchedule.td
@@ -81,6 +81,11 @@ class SchedMachineModel {
int LoadLatency = -1; // Cycles for loads to access the cache.
int HighLatency = -1; // Approximation of cycles for "high latency" ops.
int MispredictPenalty = -1; // Extra cycles for a mispredicted branch.
+ // Extra cycles a load stalls when store-to-load forwarding fails. Left unset
+ // (-1), targets fall back to a conservative default (see
+ // MCSchedModel::DefaultStoreLoadForwardingPenalty); a target may override
+ // with a more precise value.
+ int StoreLoadForwardingPenalty = -1;
// Per-cycle resources tables.
ProcessorItineraries Itineraries = NoItineraries;
diff --git a/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h b/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h
index 538d62626b37b9..b35a49c2274873 100644
--- a/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h
+++ b/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h
@@ -19,6 +19,7 @@
#define LLVM_TRANSFORMS_VECTORIZE_SLPVECTORIZER_H
#include "llvm/ADT/ArrayRef.h"
+#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/MapVector.h"
#include "llvm/ADT/SetVector.h"
#include "llvm/ADT/SmallVector.h"
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index 1f13106214ea9c..f5e57ea7c4d2a1 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -1924,20 +1924,17 @@ bool MemoryDepChecker::couldPreventStoreLoadForward(uint64_t Distance,
// place. Vectorizing in such cases does not make sense.
// Store-load forwarding distance.
- // After this many iterations store-to-load forwarding conflicts should not
- // cause any slowdowns.
- const uint64_t NumItersForStoreLoadThroughMemory = 8 * TypeByteSize;
// Maximum vector factor.
uint64_t MaxVFWithoutSLForwardIssuesPowerOf2 =
std::min(VectorizerParams::MaxVectorWidth * TypeByteSize,
MaxStoreLoadForwardSafeDistanceInBits);
- // Compute the smallest VF at which the store and load would be misaligned.
+ // Compute the smallest VF at which the store and load would be misaligned
+ // and recent enough to still be in the store buffer. The predicate is
+ // shared with SLP via MemoryDepChecker::isStoreLoadForwardingConflict.
for (uint64_t VF = 2 * TypeByteSize;
VF <= MaxVFWithoutSLForwardIssuesPowerOf2; VF *= 2) {
- // If the number of vector iteration between the store and the load are
- // small we could incur conflicts.
- if (Distance % VF && Distance / VF < NumItersForStoreLoadThroughMemory) {
+ if (isStoreLoadForwardingConflict(Distance, VF, TypeByteSize)) {
MaxVFWithoutSLForwardIssuesPowerOf2 = (VF >> 1);
break;
}
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index 595c2d10dd1188..b4a57d1c006354 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -38,6 +38,12 @@ static cl::opt<unsigned> CacheLineSize(
cl::desc("Use this to override the target cache line size when "
"specified by the user."));
+static cl::opt<int> StoreLoadForwardingConflictCost(
+ "store-load-forwarding-conflict-cost", cl::init(-1), cl::Hidden,
+ cl::desc("Override the per-occurrence cost the SLP vectorizer adds for a "
+ "store-to-load forwarding conflict (negative = use target "
+ "value)."));
+
static cl::opt<unsigned> MinPageSize(
"min-page-size", cl::init(0), cl::Hidden,
cl::desc("Use this to override the target's minimum page size."));
@@ -881,6 +887,14 @@ unsigned TargetTransformInfo::getCacheLineSize() const {
: TTIImpl->getCacheLineSize();
}
+InstructionCost TargetTransformInfo::getStoreLoadForwardingConflictCost(
+ Type *VecTy, TargetCostKind CostKind) const {
+ // A negative override means "unset"; fall back to the target's value.
+ if (StoreLoadForwardingConflictCost >= 0)
+ return InstructionCost(StoreLoadForwardingConflictCost);
+ return TTIImpl->getStoreLoadForwardingConflictCost(VecTy, CostKind);
+}
+
std::optional<unsigned>
TargetTransformInfo::getCacheSize(CacheLevel Level) const {
return TTIImpl->getCacheSize(Level);
diff --git a/llvm/lib/MC/MCSchedule.cpp b/llvm/lib/MC/MCSchedule.cpp
index e2124c1c76355f..442e2067bc2e70 100644
--- a/llvm/lib/MC/MCSchedule.cpp
+++ b/llvm/lib/MC/MCSchedule.cpp
@@ -44,6 +44,7 @@ const MCSchedModel MCSchedModel::Default = {DefaultIssueWidth,
DefaultLoadLatency,
DefaultHighLatency,
DefaultMispredictPenalty,
+ DefaultStoreLoadForwardingPenalty,
false,
true,
/*EnableIntervals=*/false,
diff --git a/llvm/lib/Target/X86/X86ScheduleZnver4.td b/llvm/lib/Target/X86/X86ScheduleZnver4.td
index ac4d31de8dbfe2..31d640fdd043b8 100644
--- a/llvm/lib/Target/X86/X86ScheduleZnver4.td
+++ b/llvm/lib/Target/X86/X86ScheduleZnver4.td
@@ -53,6 +53,10 @@ def Znver4Model : SchedMachineModel {
// The branch misprediction penalty is in the range from 11 to 18 cycles,
// <...>. The common case penalty is 13 cycles.
let MispredictPenalty = 13;
+ // AMD SOG Zen4, 2.12: a failed store-to-load forward costs roughly a full
+ // L1 load latency of extra delay while the store drains. Used by the SLP
+ // vectorizer to price store chains that would break forwarding.
+ let StoreLoadForwardingPenalty = 13;
let PostRAScheduler = 1; // Enable Post RegAlloc Scheduler pass.
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 069facf6fb4e0e..fe5c1b7338e581 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -329,6 +329,11 @@ static cl::opt<unsigned> SLPRuntimeAliasChecksMaxScalarCostPercent(
"guarded scalar region cost, before versioning is rejected to "
"avoid pessimizing the scalar fallback path."));
+static cl::opt<bool> EnableSLPStoreLoadForwardCheck(
+ "slp-store-load-forward-check", cl::init(true), cl::Hidden,
+ cl::desc("Add a cost penalty to store chains whose vectorization would "
+ "break store-to-load forwarding in SLP"));
+
// Limit the number of alias checks. The limit is chosen so that
// it has no negative effect on the llvm benchmarks.
static const unsigned AliasedCheckLimit = 10;
@@ -1005,6 +1010,13 @@ class slpvectorizer::BoUpSLP {
/// holding live values over call sites.
InstructionCost getSpillCost();
+ /// \returns true if widening the store chain anchored at \p BaseStore into a
+ /// vector store of \p VF elements would break store-to-load forwarding for a
+ /// nearby loop-carried load (a short, misaligned backward dependence). Used
+ /// to add an STLF penalty to the store entry's cost. Result is memoized in
+ /// StlfConflictCache.
+ bool findStoreLoadForwardingConflict(StoreInst *BaseStore, unsigned VF);
+
/// Calculates the cost of the subtrees, trims non-profitable ones and returns
/// final cost.
InstructionCost
@@ -1240,6 +1252,9 @@ class slpvectorizer::BoUpSLP {
TreeEntryToStridedPtrInfoMap.clear();
CurrentLoopNest.clear();
MergedLoopBTCs.clear();
+ // Loads/stores may be reconstructed on the next vectorization attempt, so
+ // stale STLF decisions must not carry over.
+ StlfConflictCache.clear();
}
unsigned getTreeSize() const { return VectorizableTree.size(); }
@@ -5685,6 +5700,11 @@ class slpvectorizer::BoUpSLP {
const DataLayout *DL;
OptimizationRemarkEmitter *ORE;
+ /// Cached STLF conflict decisions keyed by (base store of chain, VF), to
+ /// avoid re-walking the tree when the store entry is costed repeatedly.
+ /// Cleared on each buildTree() via deleteTree().
+ SmallDenseMap<std::pair<const StoreInst *, unsigned>, bool> StlfConflictCache;
+
unsigned MaxVecRegSize; // This is set by TTI or overridden by cl::opt.
unsigned MinVecRegSize; // Set by cl::opt (default: 128).
@@ -17721,6 +17741,14 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
BaseSI->getPointerAddressSpace(), CostKind, OpInfo);
}
}
+ // Widening this store chain can break store-to-load forwarding for a
+ // nearby loop-carried load. Rather than reject the tree outright, add
+ // the target's modeled STLF penalty so a chain that is still profitable
+ // after paying it can vectorize.
+ if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
+ !E->getInterleaveFactor() &&
+ findStoreLoadForwardingConflict(BaseSI, E->getVectorFactor()))
+ VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
return VecStCost + CommonCost;
};
SmallVector<Value *> PointerOps(VL.size());
@@ -28257,6 +28285,109 @@ bool SLPVectorizerPass::runImpl(Function &F, ScalarEvolution *SE_,
return Changed;
}
+bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
+ unsigned VF) {
+ if (!BaseStore)
+ return false;
+
+ StoreInst *FirstStore = BaseStore;
+
+ // Memoize per (store, VF); the entry is re-costed repeatedly.
+ auto Key = std::make_pair(FirstStore, VF);
+ auto CacheIt = StlfConflictCache.find(Key);
+ if (CacheIt != StlfConflictCache.end())
+ return CacheIt->second;
+
+ auto CacheAndReturn = [&](bool Result) -> bool {
+ StlfConflictCache[Key] = Result;
+ // No conflict at VF implies none at any smaller power-of-2 VF, so seed
+ // those entries too. Conflicts do not propagate downward.
+ if (!Result && isPowerOf2_32(VF))
+ for (unsigned V = VF / 2; V >= 2; V /= 2)
+ StlfConflictCache.try_emplace(std::make_pair(FirstStore, V), false);
+ return Result;
+ };
+
+ Loop *L = LI->getLoopFor(FirstStore->getParent());
+ if (!L)
+ return CacheAndReturn(false);
+
+ Type *ValueTy = FirstStore->getValueOperand()->getType();
+ TypeSize StoreSize = DL->getTypeStoreSize(ValueTy);
+ if (StoreSize.isScalable())
+ return CacheAndReturn(false);
+ uint64_t ElementSize = StoreSize.getFixedValue();
+ if (ElementSize == 0)
+ return CacheAndReturn(false);
+ uint64_t VectorStoreBytes = uint64_t(VF) * ElementSize;
+ LLVM_DEBUG(dbgs() << "SLP: STLF check: VF=" << VF
+ << " ElementSize=" << ElementSize
+ << " VectorStoreBytes=" << VectorStoreBytes << "\n");
+
+ // Enumerate candidate loads from the tree, not the whole loop. A conflicting
+ // load feeds the vectorized nodes, so it is reachable by walking operands
+ // from the tree scalars (including through gather leaves such as splats).
+ // Stay inside the loop to bound the walk by the tree's cone.
+ Value *StoreBase = getUnderlyingObject(FirstStore->getPointerOperand());
+ SmallPtrSet<LoadInst *, 8> CandidateLoads;
+ SmallPtrSet<const Value *, 32> Visited;
+ SmallVector<Value *, 32> Worklist;
+ for (const std::unique_ptr<TreeEntry> &TEPtr : VectorizableTree) {
+ const TreeEntry *TE = TEPtr.get();
+ if (DeletedNodes.contains(TE))
+ continue;
+ Worklist.append(TE->Scalars.begin(), TE->Scalars.end());
+ }
+ while (!Worklist.empty()) {
+ Value *V = Worklist.pop_back_val();
+ if (!Visited.insert(V).second)
+ continue;
+ if (auto *LoadI = dyn_cast<LoadInst>(V)) {
+ if (LoadI->isSimple() &&
+ getUnderlyingObject(LoadI->getPointerOperand()) == StoreBase)
+ CandidateLoads.insert(LoadI);
+ continue;
+ }
+ auto *I = dyn_cast<Instruction>(V);
+ if (!I || !L->contains(I))
+ continue;
+ Worklist.append(I->op_begin(), I->op_end());
+ }
+
+ if (CandidateLoads.empty())
+ return CacheAndReturn(false);
+
+ // For each candidate load, the widened chain becomes one wide store at the
+ // base; check whether the load straddles two such wide stores.
+ for (LoadInst *LoadI : CandidateLoads) {
+ std::optional<int64_t> Diff =
+ getPointersDiff(ValueTy, FirstStore->getPointerOperand(),
+ LoadI->getType(), LoadI->getPointerOperand(), *DL, *SE,
+ /*StrictCheck=*/true, /*CheckType=*/false);
+ if (!Diff || *Diff >= 0)
+ continue;
+
+ uint64_t Distance = -static_cast<uint64_t>(*Diff) * ElementSize;
+ LLVM_DEBUG(dbgs() << "SLP: STLF: load=" << *LoadI << " distance="
+ << Distance << " bytes from chain base\n");
+
+ // Conflict if the load is misaligned to the wide store within the recency
+ // window.
+ if (MemoryDepChecker::isStoreLoadForwardingConflict(
+ Distance, VectorStoreBytes, ElementSize)) {
+ LLVM_DEBUG(dbgs() << "SLP: Store-load forwarding conflict: "
+ << (isVectorized(LoadI) ? "widened" : "scalar")
+ << " load, distance " << Distance
+ << " bytes, vector store width " << VectorStoreBytes
+ << " bytes, misalignment "
+ << (Distance % VectorStoreBytes) << "\n");
+ return CacheAndReturn(true);
+ }
+ }
+
+ return CacheAndReturn(false);
+}
+
std::optional<bool>
SLPVectorizerPass::vectorizeStoreChain(ArrayRef<Value *> Chain, BoUpSLP &R,
unsigned Idx, unsigned MinVF,
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index b705f5cea5347a..906c560635fc02 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -1,5 +1,12 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
-; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 | FileCheck %s
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-ON
+; RUN: opt < %s -passes=slp-vectorizer -slp-store-load-forward-check=false -S \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-OFF
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
;
; Test 1: Backward loop-carried load at misaligned distance.
@@ -16,31 +23,58 @@
; store at A[i]. The load at A[i-5] is 20 bytes behind:
; 20 % 16 = 4 -> misaligned, load straddles two pending vector stores
; 20 / 16 = 1 -> store still hot in store buffer
-; The new check should bail out and leave the four-wide chain scalar.
+; The STLF penalty makes the VF=4 store entry unprofitable. Sub-chains at
+; VF=2 are priced per-base: {A[i+1],A[i+2]} has distance 24, 24%8=0 → no
+; conflict, no penalty, so it still vectorizes.
;
define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
-;
-;
-;
-; CHECK-LABEL: define void @stlf_conflict_backward_misaligned(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
-; CHECK-NEXT: [[ENTRY:.*]]:
-; CHECK-NEXT: br label %[[FOR_BODY:.*]]
-; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 5, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
-; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; CHECK: [[FOR_END]]:
-; CHECK-NEXT: ret void
+; STLF-ON-LABEL: define void @stlf_conflict_backward_misaligned(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 5, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
+; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
+; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
+; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
+; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_conflict_backward_misaligned(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 5, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-OFF-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
;
entry:
br label %for.body
@@ -92,9 +126,8 @@ for.end:
; The check should NOT fire; vectorization is allowed.
;
define void @stlf_no_conflict_backward_aligned(ptr noalias %A, i64 %n) {
-;
; CHECK-LABEL: define void @stlf_no_conflict_backward_aligned(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: br label %[[FOR_BODY:.*]]
; CHECK: [[FOR_BODY]]:
@@ -157,7 +190,6 @@ for.end:
; Vectorization should proceed.
;
define void @stlf_no_conflict_forward(ptr noalias %A, i64 %n) {
-;
; CHECK-LABEL: define void @stlf_no_conflict_forward(
; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
; CHECK-NEXT: [[ENTRY:.*]]:
@@ -222,7 +254,6 @@ for.end:
; store buffer is hot. Outside loops the check should not fire.
;
define void @stlf_no_conflict_no_loop(ptr noalias %A) {
-;
; CHECK-LABEL: define void @stlf_no_conflict_no_loop(
; CHECK-SAME: ptr noalias [[A:%.*]]) #[[ATTR0]] {
; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 -5
@@ -260,7 +291,6 @@ define void @stlf_no_conflict_no_loop(ptr noalias %A) {
; widened store, so the check should not fire.
;
define void @stlf_no_conflict_different_base(ptr noalias %A, ptr noalias %B,
-;
; CHECK-LABEL: define void @stlf_no_conflict_different_base(
; CHECK-SAME: ptr noalias [[A:%.*]], ptr noalias [[B:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
; CHECK-NEXT: [[ENTRY:.*]]:
@@ -327,7 +357,6 @@ for.end:
; finds no offending load and vectorization proceeds.
;
define void @stlf_volatile_load_skipped(ptr noalias %A, i64 %n) {
-;
; CHECK-LABEL: define void @stlf_volatile_load_skipped(
; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
; CHECK-NEXT: [[ENTRY:.*]]:
@@ -390,35 +419,67 @@ for.end:
;
; Both A[i-5] (distance 20 bytes) and A[i-7] (distance 28 bytes) are
; misaligned to a 16-byte vector store and within the safety window. The
-; check fires on the first one encountered and bails out.
+; check fires on the first one encountered and the store entry is penalized,
+; keeping the chain scalar.
;
define void @stlf_multiple_conflicting_loads(ptr noalias %A, i64 %n) {
-;
-;
-;
-; CHECK-LABEL: define void @stlf_multiple_conflicting_loads(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; CHECK-NEXT: [[ENTRY:.*]]:
-; CHECK-NEXT: br label %[[FOR_BODY:.*]]
-; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
-; CHECK-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
-; CHECK-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
-; CHECK-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
-; CHECK-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
-; CHECK-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
-; CHECK-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
-; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[SUM]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; CHECK: [[FOR_END]]:
-; CHECK-NEXT: ret void
+; STLF-ON-LABEL: define void @stlf_multiple_conflicting_loads(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-ON-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
+; STLF-ON-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
+; STLF-ON-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
+; STLF-ON-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
+; STLF-ON-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
+; STLF-ON-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[SUM]], 1
+; STLF-ON-NEXT: [[T2:%.*]] = add nsw i32 [[SUM]], 2
+; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[SUM]], 3
+; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[SUM]], 4
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: store i32 [[T2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP2]], align 4
+; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_multiple_conflicting_loads(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-OFF-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
+; STLF-OFF-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
+; STLF-OFF-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
+; STLF-OFF-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
+; STLF-OFF-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
+; STLF-OFF-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[SUM]], i64 0
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
;
entry:
br label %for.body
@@ -471,7 +532,6 @@ for.end:
; conflict, and vectorization proceeds.
;
define void @stlf_pointer_diff_unknown(ptr noalias %A, ptr noalias %IdxA, i64 %n) {
-;
; CHECK-LABEL: define void @stlf_pointer_diff_unknown(
; CHECK-SAME: ptr noalias [[A:%.*]], ptr noalias [[IDXA:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
; CHECK-NEXT: [[ENTRY:.*]]:
@@ -532,35 +592,62 @@ for.end:
}
;
-; Test 9: Distance < VectorStoreBytes (load fully within previous vector
-; store).
-;
-; The backward load is only 1 element (4 bytes) before the stores; the entire
-; load lies inside the previous iteration's 16-byte vector store, so STLF
-; can forward. The check's `Distance < VectorStoreBytes` early-continue path
-; catches this and allows vectorization.
-;
-define void @stlf_no_conflict_within_prev_store(ptr noalias %A, i64 %n) {
-;
-; CHECK-LABEL: define void @stlf_no_conflict_within_prev_store(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; CHECK-NEXT: [[ENTRY:.*]]:
-; CHECK-NEXT: br label %[[FOR_BODY:.*]]
-; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
-; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; CHECK: [[FOR_END]]:
-; CHECK-NEXT: ret void
+; Test 9: Motivating example — a[i] = a[i-1] + 1.
+;
+; Distance = 4 bytes (1 element) from chain base. VectorStoreBytes = 16
+; at VF=4. 4 % 16 = 4 → misaligned → STLF conflict. The full 4-wide store
+; entry is penalized and is not profitable. Sub-chains that are individually
+; misaligned are penalized too; only sub-chains where the distance happens to
+; be aligned to the narrower vector store width vectorize.
+;
+define void @stlf_conflict_short_backward(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @stlf_conflict_short_backward(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
+; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
+; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
+; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
+; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
+; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_conflict_short_backward(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
+; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-OFF-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
;
entry:
br label %for.body
@@ -600,79 +687,7 @@ for.end:
}
;
-; Test 10: Loop body too large -> STLF check skipped.
-;
-; Same misaligned-distance shape as Test 1, but the third RUN line lowers
-; the SLP loop-size threshold to 10. The loop body is ~20 instructions
-; (well above 10 yet well below the default 128). With the default
-; threshold of 128 the check still runs and rejects vectorization, with
-; the STLF flag forced off vectorization is allowed, and with the
-; threshold lowered to 10 the loop is treated as large so the check is
-; skipped and vectorization is allowed.
-;
-define void @stlf_large_loop_bypassed(ptr noalias %A, i64 %n) {
-;
-;
-;
-; CHECK-LABEL: define void @stlf_large_loop_bypassed(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; CHECK-NEXT: [[ENTRY:.*]]:
-; CHECK-NEXT: br label %[[FOR_BODY:.*]]
-; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 5, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
-; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; CHECK: [[FOR_END]]:
-; CHECK-NEXT: ret void
-;
-entry:
- br label %for.body
-
-for.body:
- %i = phi i64 [ 5, %entry ], [ %i.next, %for.body ]
-
- %back.idx = sub i64 %i, 5
- %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
- %t = load i32, ptr %back.gep, align 4
-
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
-
- %i1 = add nuw nsw i64 %i, 1
- %i2 = add nuw nsw i64 %i, 2
- %i3 = add nuw nsw i64 %i, 3
-
- %gep0 = getelementptr inbounds i32, ptr %A, i64 %i
- %gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
- %gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
- %gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
- store i32 %t1, ptr %gep0, align 4
- store i32 %t2, ptr %gep1, align 4
- store i32 %t3, ptr %gep2, align 4
- store i32 %t4, ptr %gep3, align 4
-
- %i.next = add nuw nsw i64 %i, 4
- %cmp = icmp slt i64 %i.next, %n
- br i1 %cmp, label %for.body, label %for.end
-
-for.end:
- ret void
-}
-
-;
-; Test 11: Distance just past NumItersForSafety boundary.
+; Test 10: Distance just past NumItersForSafety boundary.
;
; For i32 with VF=4: ElementSize=4, VectorStoreBytes=16,
; NumItersForSafety = 8 * ElementSize = 32. The check fires only when
@@ -682,7 +697,6 @@ for.end:
; (misaligned). Vectorization proceeds.
;
define void @stlf_no_conflict_iter_safety_boundary(ptr noalias %A, i64 %n) {
-;
; CHECK-LABEL: define void @stlf_no_conflict_iter_safety_boundary(
; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
; CHECK-NEXT: [[ENTRY:.*]]:
@@ -741,7 +755,7 @@ for.end:
}
;
-; Test 12: Non-i32 element type (i64 VF=2).
+; Test 11: Non-i32 element type (i64 VF=2).
;
; Verifies the byte-distance math works for wider types. With i64
; (ElementSize=8) and VF=2: VectorStoreBytes=16, NumItersForSafety=64.
@@ -749,28 +763,47 @@ for.end:
; 24%16=8 != 0, so the check fires.
;
define void @stlf_conflict_backward_misaligned_i64(ptr noalias %A, i64 %n) {
-;
-;
-;
-; CHECK-LABEL: define void @stlf_conflict_backward_misaligned_i64(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; CHECK-NEXT: [[ENTRY:.*]]:
-; CHECK-NEXT: br label %[[FOR_BODY:.*]]
-; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 3, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 3
-; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[BACK_IDX]]
-; CHECK-NEXT: [[T:%.*]] = load i64, ptr [[BACK_GEP]], align 8
-; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <2 x i64> poison, i64 [[T]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <2 x i64> [[TMP0]], <2 x i64> poison, <2 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <2 x i64> [[TMP1]], <i64 1, i64 2>
-; CHECK-NEXT: store <2 x i64> [[TMP2]], ptr [[GEP0]], align 8
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 2
-; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; CHECK: [[FOR_END]]:
-; CHECK-NEXT: ret void
+; STLF-ON-LABEL: define void @stlf_conflict_backward_misaligned_i64(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 3, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 3
+; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-ON-NEXT: [[T:%.*]] = load i64, ptr [[BACK_GEP]], align 8
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i64 [[T]], 1
+; STLF-ON-NEXT: [[T2:%.*]] = add nsw i64 [[T]], 2
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: store i64 [[T1]], ptr [[GEP0]], align 8
+; STLF-ON-NEXT: store i64 [[T2]], ptr [[GEP1]], align 8
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_conflict_backward_misaligned_i64(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 3, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 3
+; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-OFF-NEXT: [[T:%.*]] = load i64, ptr [[BACK_GEP]], align 8
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <2 x i64> poison, i64 [[T]], i64 0
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <2 x i64> [[TMP0]], <2 x i64> poison, <2 x i32> zeroinitializer
+; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <2 x i64> [[TMP1]], <i64 1, i64 2>
+; STLF-OFF-NEXT: store <2 x i64> [[TMP2]], ptr [[GEP0]], align 8
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 2
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
;
entry:
br label %for.body
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
new file mode 100644
index 00000000000000..d371948c3eb6b7
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
@@ -0,0 +1,105 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; Store-to-load forwarding (STLF) is priced into the SLP cost model rather than
+; vetoing vectorization outright. The motivating loop a[i] = a[i-1] + 1 has a
+; misaligned backward dependence that breaks forwarding once the four scalar
+; stores become one 16-byte vector store.
+;
+; With the target's default STLF penalty the small tree is not worth the stall,
+; so it stays (mostly) scalar. Zeroing the penalty via the tuning knob lets the
+; very same tree vectorize to <4 x i32> -- demonstrating this is a cost, not a
+; hard bail-out, so a sufficiently profitable tree can still vectorize.
+;
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefix=PENALTY
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: -store-load-forwarding-conflict-cost=0 \
+; RUN: | FileCheck %s --check-prefix=NOPENALTY
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @stlf_recurrence(ptr noalias %A, i64 %n) {
+; PENALTY-LABEL: define void @stlf_recurrence(
+; PENALTY-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; PENALTY-NEXT: [[ENTRY:.*]]:
+; PENALTY-NEXT: br label %[[FOR_BODY:.*]]
+; PENALTY: [[FOR_BODY]]:
+; PENALTY-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; PENALTY-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
+; PENALTY-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; PENALTY-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; PENALTY-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
+; PENALTY-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
+; PENALTY-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; PENALTY-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; PENALTY-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; PENALTY-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; PENALTY-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; PENALTY-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; PENALTY-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
+; PENALTY-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
+; PENALTY-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
+; PENALTY-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; PENALTY-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; PENALTY-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; PENALTY-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; PENALTY-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; PENALTY: [[FOR_END]]:
+; PENALTY-NEXT: ret void
+;
+; NOPENALTY-LABEL: define void @stlf_recurrence(
+; NOPENALTY-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; NOPENALTY-NEXT: [[ENTRY:.*]]:
+; NOPENALTY-NEXT: br label %[[FOR_BODY:.*]]
+; NOPENALTY: [[FOR_BODY]]:
+; NOPENALTY-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; NOPENALTY-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
+; NOPENALTY-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; NOPENALTY-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; NOPENALTY-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; NOPENALTY-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
+; NOPENALTY-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; NOPENALTY-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; NOPENALTY-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; NOPENALTY-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; NOPENALTY-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; NOPENALTY-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; NOPENALTY: [[FOR_END]]:
+; NOPENALTY-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 1, %entry ], [ %i.next, %for.body ]
+
+ %back.idx = sub i64 %i, 1
+ %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
+ %t = load i32, ptr %back.gep, align 4
+
+ %t1 = add nsw i32 %t, 1
+ %t2 = add nsw i32 %t, 2
+ %t3 = add nsw i32 %t, 3
+ %t4 = add nsw i32 %t, 4
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+
+ %gep0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
+ %gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
+ %gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
+
+ store i32 %t1, ptr %gep0, align 4
+ store i32 %t2, ptr %gep1, align 4
+ store i32 %t3, ptr %gep2, align 4
+ store i32 %t4, ptr %gep3, align 4
+
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
new file mode 100644
index 00000000000000..4b22025a1f4bd1
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
@@ -0,0 +1,145 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; A store chain that DOES break store-to-load forwarding, but whose stored
+; values come from a long, isomorphic, vectorizable compute chain over the
+; recurrence load. The value-subtree savings from widening (many scalar ops ->
+; a few vector ops) exceed the modeled STLF stall, so the cost model vectorizes
+; it anyway. This is the case a hard veto would wrongly discard: a veto is the
+; infinite-penalty limit of the cost, and any chain whose vector savings beat
+; the stall should still vectorize.
+;
+; Default run: the znver5 penalty (StoreLoadForwardingPenalty = 13) is paid but
+; the whole chain still vectorizes to <4 x i32> because the value tree saves
+; more than that. The absurd-penalty run (the veto/infinite limit) prices the
+; chain out again, proving the store entry really is charged the STLF penalty
+; (the chain is a genuine forwarding conflict, not a no-op).
+;
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefix=VEC
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: -store-load-forwarding-conflict-cost=1000 \
+; RUN: | FileCheck %s --check-prefix=VETO
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @stlf_profitable(ptr noalias %A, i64 %n) {
+; VEC-LABEL: define void @stlf_profitable(
+; VEC-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; VEC-NEXT: [[ENTRY:.*]]:
+; VEC-NEXT: br label %[[FOR_BODY:.*]]
+; VEC: [[FOR_BODY]]:
+; VEC-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; VEC-NEXT: [[BI:%.*]] = add i64 [[I]], -1
+; VEC-NEXT: [[BGEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI]]
+; VEC-NEXT: [[T:%.*]] = load i32, ptr [[BGEP]], align 4
+; VEC-NEXT: [[G0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; VEC-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
+; VEC-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; VEC-NEXT: [[TMP2:%.*]] = add <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; VEC-NEXT: [[TMP3:%.*]] = mul <4 x i32> [[TMP2]], [[TMP2]]
+; VEC-NEXT: [[TMP4:%.*]] = add <4 x i32> [[TMP3]], <i32 7, i32 8, i32 9, i32 10>
+; VEC-NEXT: [[TMP5:%.*]] = mul <4 x i32> [[TMP4]], [[TMP2]]
+; VEC-NEXT: [[TMP6:%.*]] = add <4 x i32> [[TMP5]], <i32 11, i32 12, i32 13, i32 14>
+; VEC-NEXT: [[TMP7:%.*]] = mul <4 x i32> [[TMP6]], [[TMP4]]
+; VEC-NEXT: store <4 x i32> [[TMP7]], ptr [[G0]], align 4
+; VEC-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; VEC-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; VEC-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; VEC: [[FOR_END]]:
+; VEC-NEXT: ret void
+;
+; VETO-LABEL: define void @stlf_profitable(
+; VETO-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; VETO-NEXT: [[ENTRY:.*]]:
+; VETO-NEXT: br label %[[FOR_BODY:.*]]
+; VETO: [[FOR_BODY]]:
+; VETO-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; VETO-NEXT: [[BI:%.*]] = add i64 [[I]], -1
+; VETO-NEXT: [[BGEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI]]
+; VETO-NEXT: [[T:%.*]] = load i32, ptr [[BGEP]], align 4
+; VETO-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
+; VETO-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
+; VETO-NEXT: [[TMP2:%.*]] = add <2 x i32> [[TMP1]], <i32 1, i32 4>
+; VETO-NEXT: [[TMP3:%.*]] = mul <2 x i32> [[TMP2]], [[TMP2]]
+; VETO-NEXT: [[TMP4:%.*]] = add <2 x i32> [[TMP3]], <i32 7, i32 10>
+; VETO-NEXT: [[TMP5:%.*]] = mul <2 x i32> [[TMP4]], [[TMP2]]
+; VETO-NEXT: [[TMP6:%.*]] = add <2 x i32> [[TMP5]], <i32 11, i32 14>
+; VETO-NEXT: [[TMP7:%.*]] = mul <2 x i32> [[TMP6]], [[TMP4]]
+; VETO-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; VETO-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; VETO-NEXT: [[G0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; VETO-NEXT: [[G1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; VETO-NEXT: [[G3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; VETO-NEXT: [[TMP8:%.*]] = extractelement <2 x i32> [[TMP7]], i64 0
+; VETO-NEXT: store i32 [[TMP8]], ptr [[G0]], align 4
+; VETO-NEXT: [[TMP9:%.*]] = add <2 x i32> [[TMP1]], <i32 2, i32 3>
+; VETO-NEXT: [[TMP10:%.*]] = mul <2 x i32> [[TMP9]], [[TMP9]]
+; VETO-NEXT: [[TMP11:%.*]] = add <2 x i32> [[TMP10]], <i32 8, i32 9>
+; VETO-NEXT: [[TMP12:%.*]] = mul <2 x i32> [[TMP11]], [[TMP9]]
+; VETO-NEXT: [[TMP13:%.*]] = add <2 x i32> [[TMP12]], <i32 12, i32 13>
+; VETO-NEXT: [[TMP14:%.*]] = mul <2 x i32> [[TMP13]], [[TMP11]]
+; VETO-NEXT: store <2 x i32> [[TMP14]], ptr [[G1]], align 4
+; VETO-NEXT: [[TMP15:%.*]] = extractelement <2 x i32> [[TMP7]], i64 1
+; VETO-NEXT: store i32 [[TMP15]], ptr [[G3]], align 4
+; VETO-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; VETO-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; VETO-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; VETO: [[FOR_END]]:
+; VETO-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 1, %entry ], [ %i.next, %for.body ]
+ %bi = add i64 %i, -1
+ %bgep = getelementptr inbounds i32, ptr %A, i64 %bi
+ %t = load i32, ptr %bgep, align 4
+
+ %a0 = add i32 %t, 1
+ %b0 = mul i32 %a0, %a0
+ %c0 = add i32 %b0, 7
+ %d0 = mul i32 %c0, %a0
+ %e0 = add i32 %d0, 11
+ %f0 = mul i32 %e0, %c0
+
+ %a1 = add i32 %t, 2
+ %b1 = mul i32 %a1, %a1
+ %c1 = add i32 %b1, 8
+ %d1 = mul i32 %c1, %a1
+ %e1 = add i32 %d1, 12
+ %f1 = mul i32 %e1, %c1
+
+ %a2 = add i32 %t, 3
+ %b2 = mul i32 %a2, %a2
+ %c2 = add i32 %b2, 9
+ %d2 = mul i32 %c2, %a2
+ %e2 = add i32 %d2, 13
+ %f2 = mul i32 %e2, %c2
+
+ %a3 = add i32 %t, 4
+ %b3 = mul i32 %a3, %a3
+ %c3 = add i32 %b3, 10
+ %d3 = mul i32 %c3, %a3
+ %e3 = add i32 %d3, 14
+ %f3 = mul i32 %e3, %c3
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+ %g0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %g1 = getelementptr inbounds i32, ptr %A, i64 %i1
+ %g2 = getelementptr inbounds i32, ptr %A, i64 %i2
+ %g3 = getelementptr inbounds i32, ptr %A, i64 %i3
+ store i32 %f0, ptr %g0, align 4
+ store i32 %f1, ptr %g1, align 4
+ store i32 %f2, ptr %g2, align 4
+ store i32 %f3, ptr %g3, align 4
+
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
diff --git a/llvm/utils/TableGen/SubtargetEmitter.cpp b/llvm/utils/TableGen/SubtargetEmitter.cpp
index 226f8068626da9..d55aad715fd080 100644
--- a/llvm/utils/TableGen/SubtargetEmitter.cpp
+++ b/llvm/utils/TableGen/SubtargetEmitter.cpp
@@ -1544,6 +1544,7 @@ void SubtargetEmitter::emitProcessorModels(raw_ostream &OS) {
emitProcessorProp(OS, PM.ModelDef, "LoadLatency", ',');
emitProcessorProp(OS, PM.ModelDef, "HighLatency", ',');
emitProcessorProp(OS, PM.ModelDef, "MispredictPenalty", ',');
+ emitProcessorProp(OS, PM.ModelDef, "StoreLoadForwardingPenalty", ',');
bool PostRAScheduler =
(PM.ModelDef ? PM.ModelDef->getValueAsBit("PostRAScheduler") : false);
>From 6766960ddae4347a14f7aea98f43f243734ef3d1 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Thu, 6 Aug 2026 15:05:19 +0530
Subject: [PATCH 02/21] [SLP] Address review comments on STLF cost modeling -
Enumerate candidate loads directly from the tree's load/gather nodes
instead of a whole-tree operand walk. - Move the scalable-type bailout early
and drop the redundant VF cast. - Extend the no-conflict cache seeding to all
divisors of VF (covers non-power-of-2 VFs), and add a
-slp-vectorize-non-power-of-2 test. - Define the fallback STLF penalty as an
InstructionCost in the TTI layer and drop the MCSchedule.h include from
TargetTransformInfoImpl.h.
---
.../llvm/Analysis/TargetTransformInfoImpl.h | 10 +-
.../Transforms/Vectorize/SLPVectorizer.cpp | 58 ++++----
.../X86/store-load-forward-conflict.ll | 92 +++++--------
.../X86/store-load-forward-non-power-of-2.ll | 125 ++++++++++++++++++
4 files changed, 187 insertions(+), 98 deletions(-)
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 2d402429a8b8a6..f27c20776c24fd 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -23,7 +23,6 @@
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Operator.h"
#include "llvm/IR/PatternMatch.h"
-#include "llvm/MC/MCSchedule.h"
#include <optional>
#include <utility>
@@ -684,10 +683,11 @@ class LLVM_ABI TargetTransformInfoImplBase {
virtual InstructionCost getStoreLoadForwardingConflictCost(
Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
- // No subtarget scheduling model is available here, so fall back to the
- // conservative default STLF stall penalty (targets with a real model go
- // through BasicTTIImpl and use their own value instead).
- return InstructionCost(MCSchedModel::DefaultStoreLoadForwardingPenalty);
+ // No subtarget scheduling model is available here, so fall back to a
+ // conservative default STLF stall penalty (~10 cycles). Targets with a real
+ // scheduling model go through BasicTTIImpl and use their own value instead.
+ constexpr unsigned DefaultStoreLoadForwardingConflictCost = 10;
+ return InstructionCost(DefaultStoreLoadForwardingConflictCost);
}
virtual std::optional<unsigned>
getCacheSize(TargetTransformInfo::CacheLevel Level) const {
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index fe5c1b7338e581..2f1dc8d58c67a1 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28300,18 +28300,20 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
auto CacheAndReturn = [&](bool Result) -> bool {
StlfConflictCache[Key] = Result;
- // No conflict at VF implies none at any smaller power-of-2 VF, so seed
- // those entries too. Conflicts do not propagate downward.
- if (!Result && isPowerOf2_32(VF))
- for (unsigned V = VF / 2; V >= 2; V /= 2)
- StlfConflictCache.try_emplace(std::make_pair(FirstStore, V), false);
+ // No conflict at VF implies none at any smaller VF that divides VF, so seed
+ // those entries too. The recency window is VF-independent (so it stays
+ // satisfied for any smaller width), and the distance remains a multiple of
+ // V * ElementSize only for divisors of VF. This covers the power-of-2
+ // ladder (each smaller power of 2 divides the larger) and any
+ // non-power-of-2 VF the caller may probe. Conflicts do not propagate
+ // downward.
+ if (!Result)
+ for (unsigned V = 2; V < VF; ++V)
+ if (VF % V == 0)
+ StlfConflictCache.try_emplace(std::make_pair(FirstStore, V), false);
return Result;
};
- Loop *L = LI->getLoopFor(FirstStore->getParent());
- if (!L)
- return CacheAndReturn(false);
-
Type *ValueTy = FirstStore->getValueOperand()->getType();
TypeSize StoreSize = DL->getTypeStoreSize(ValueTy);
if (StoreSize.isScalable())
@@ -28319,39 +28321,33 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
uint64_t ElementSize = StoreSize.getFixedValue();
if (ElementSize == 0)
return CacheAndReturn(false);
- uint64_t VectorStoreBytes = uint64_t(VF) * ElementSize;
+
+ // Store-to-load forwarding hazards are a loop-carried concern.
+ if (!LI->getLoopFor(FirstStore->getParent()))
+ return CacheAndReturn(false);
+
+ uint64_t VectorStoreBytes = VF * ElementSize;
LLVM_DEBUG(dbgs() << "SLP: STLF check: VF=" << VF
<< " ElementSize=" << ElementSize
<< " VectorStoreBytes=" << VectorStoreBytes << "\n");
- // Enumerate candidate loads from the tree, not the whole loop. A conflicting
- // load feeds the vectorized nodes, so it is reachable by walking operands
- // from the tree scalars (including through gather leaves such as splats).
- // Stay inside the loop to bound the walk by the tree's cone.
+ // Enumerate candidate loads directly from the tree's load and gather nodes: a
+ // conflicting load is either widened (a load node) or packed into a gather
+ // leaf (e.g. a splat), so scanning those node kinds is sufficient.
Value *StoreBase = getUnderlyingObject(FirstStore->getPointerOperand());
SmallPtrSet<LoadInst *, 8> CandidateLoads;
- SmallPtrSet<const Value *, 32> Visited;
- SmallVector<Value *, 32> Worklist;
for (const std::unique_ptr<TreeEntry> &TEPtr : VectorizableTree) {
const TreeEntry *TE = TEPtr.get();
if (DeletedNodes.contains(TE))
continue;
- Worklist.append(TE->Scalars.begin(), TE->Scalars.end());
- }
- while (!Worklist.empty()) {
- Value *V = Worklist.pop_back_val();
- if (!Visited.insert(V).second)
- continue;
- if (auto *LoadI = dyn_cast<LoadInst>(V)) {
- if (LoadI->isSimple() &&
- getUnderlyingObject(LoadI->getPointerOperand()) == StoreBase)
- CandidateLoads.insert(LoadI);
- continue;
- }
- auto *I = dyn_cast<Instruction>(V);
- if (!I || !L->contains(I))
+ if (!TE->isGather() &&
+ !(TE->hasState() && TE->getOpcode() == Instruction::Load))
continue;
- Worklist.append(I->op_begin(), I->op_end());
+ for (Value *V : TE->Scalars)
+ if (auto *LoadI = dyn_cast<LoadInst>(V))
+ if (LoadI->isSimple() &&
+ getUnderlyingObject(LoadI->getPointerOperand()) == StoreBase)
+ CandidateLoads.insert(LoadI);
}
if (CandidateLoads.empty())
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index 906c560635fc02..f17d390ee62200 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -415,71 +415,39 @@ for.end:
}
;
-; Test 7: Multiple conflicting loads in the same loop.
+; Test 7: Loads that feed scalar arithmetic below a gather/splat leaf.
;
-; Both A[i-5] (distance 20 bytes) and A[i-7] (distance 28 bytes) are
-; misaligned to a 16-byte vector store and within the safety window. The
-; check fires on the first one encountered and the store entry is penalized,
-; keeping the chain scalar.
+; A[i-5] and A[i-7] would be misaligned to a 16-byte vector store, but they
+; only feed the scalar %sum, which is then broadcast. Candidate loads are
+; enumerated from the tree's load and gather nodes, so loads sitting below a
+; gather leaf (here, under %sum) are not seen and no penalty is charged; the
+; chain vectorizes. This documents the reachability limit of the node-based
+; enumeration.
;
define void @stlf_multiple_conflicting_loads(ptr noalias %A, i64 %n) {
-; STLF-ON-LABEL: define void @stlf_multiple_conflicting_loads(
-; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-ON-NEXT: [[ENTRY:.*]]:
-; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-ON: [[FOR_BODY]]:
-; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-ON-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-ON-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
-; STLF-ON-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
-; STLF-ON-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
-; STLF-ON-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
-; STLF-ON-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
-; STLF-ON-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
-; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[SUM]], 1
-; STLF-ON-NEXT: [[T2:%.*]] = add nsw i32 [[SUM]], 2
-; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[SUM]], 3
-; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[SUM]], 4
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
-; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
-; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
-; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
-; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
-; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: store i32 [[T2]], ptr [[GEP1]], align 4
-; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP2]], align 4
-; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
-; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-ON: [[FOR_END]]:
-; STLF-ON-NEXT: ret void
-;
-; STLF-OFF-LABEL: define void @stlf_multiple_conflicting_loads(
-; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-OFF-NEXT: [[ENTRY:.*]]:
-; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-OFF: [[FOR_BODY]]:
-; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-OFF-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-OFF-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
-; STLF-OFF-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
-; STLF-OFF-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
-; STLF-OFF-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
-; STLF-OFF-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
-; STLF-OFF-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
-; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[SUM]], i64 0
-; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-OFF: [[FOR_END]]:
-; STLF-OFF-NEXT: ret void
+; CHECK-LABEL: define void @stlf_multiple_conflicting_loads(
+; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: br label %[[FOR_BODY:.*]]
+; CHECK: [[FOR_BODY]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
+; CHECK-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
+; CHECK-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
+; CHECK-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
+; CHECK-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
+; CHECK-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
+; CHECK-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
+; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[SUM]], i64 0
+; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; CHECK: [[FOR_END]]:
+; CHECK-NEXT: ret void
;
entry:
br label %for.body
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
new file mode 100644
index 00000000000000..74433ed96fdd84
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
@@ -0,0 +1,125 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=slp-vectorizer -slp-vectorize-non-power-of-2 -S \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-ON
+; RUN: opt < %s -passes=slp-vectorizer -slp-vectorize-non-power-of-2 \
+; RUN: -slp-store-load-forward-check=false -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-OFF
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+;
+; Store-to-load forwarding hazard at a non-power-of-2 vector factor.
+;
+; for (int i = 7; i < n; i += 7) {
+; int t = A[i-5]; ; load 20 bytes (5*4) before the store base
+; A[i] = t + 1;
+; ...
+; A[i+6] = t + 7;
+; }
+;
+; With -slp-vectorize-non-power-of-2, SLP first probes VF=7. The backward load
+; A[i-5] is 20 bytes behind the store base:
+; 20 % (7*4=28) = 20 -> misaligned, load straddles two pending vector stores
+; 20 / 28 = 0 -> store still hot in the store buffer
+; so the STLF penalty prices the VF=7 (and the VF=4 base) widening out. With the
+; check disabled the whole chain widens to a single <7 x i32> store. This
+; exercises the hazard check at a non-power-of-2 VF and the divisor-based cache
+; propagation.
+;
+define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @stlf_non_power_of_2(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
+; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
+; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
+; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
+; STLF-ON-NEXT: [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> poison, <4 x i32> zeroinitializer
+; STLF-ON-NEXT: [[TMP5:%.*]] = add nsw <4 x i32> [[TMP4]], <i32 4, i32 5, i32 6, i32 7>
+; STLF-ON-NEXT: store <4 x i32> [[TMP5]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 7
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_non_power_of_2(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; STLF-OFF-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <7 x i32> poison, i32 [[T]], i64 0
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <7 x i32> [[TMP0]], <7 x i32> poison, <7 x i32> zeroinitializer
+; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <7 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; STLF-OFF-NEXT: store <7 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 7
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 7, %entry ], [ %i.next, %for.body ]
+ %back.idx = sub i64 %i, 5
+ %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
+ %t = load i32, ptr %back.gep, align 4
+ %t1 = add nsw i32 %t, 1
+ %t2 = add nsw i32 %t, 2
+ %t3 = add nsw i32 %t, 3
+ %t4 = add nsw i32 %t, 4
+ %t5 = add nsw i32 %t, 5
+ %t6 = add nsw i32 %t, 6
+ %t7 = add nsw i32 %t, 7
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+ %i4 = add nuw nsw i64 %i, 4
+ %i5 = add nuw nsw i64 %i, 5
+ %i6 = add nuw nsw i64 %i, 6
+ %gep0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
+ %gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
+ %gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
+ %gep4 = getelementptr inbounds i32, ptr %A, i64 %i4
+ %gep5 = getelementptr inbounds i32, ptr %A, i64 %i5
+ %gep6 = getelementptr inbounds i32, ptr %A, i64 %i6
+ store i32 %t1, ptr %gep0, align 4
+ store i32 %t2, ptr %gep1, align 4
+ store i32 %t3, ptr %gep2, align 4
+ store i32 %t4, ptr %gep3, align 4
+ store i32 %t5, ptr %gep4, align 4
+ store i32 %t6, ptr %gep5, align 4
+ store i32 %t7, ptr %gep6, align 4
+ %i.next = add nuw nsw i64 %i, 7
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK: {{.*}}
>From e8300d5744a85326e8a64e1ceecf2c7bd566da4e Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Wed, 12 Aug 2026 13:55:36 +0530
Subject: [PATCH 03/21] [SLP] Detect wider-than-store loads in STLF check;
dedup penalty default Extend MemoryDepChecker::isStoreLoadForwardingConflict
with an optional LoadElementSize: an aligned load wider than the widened
store still straddles two wide stores, which the misalignment-only test
missed. LAA's call is unchanged (default 0), keeping it NFC; SLP passes the
load width. Also dedup the store-load-forwarding penalty default via
MCSchedModel and drop an unused DenseMap.h include.
---
.../llvm/Analysis/LoopAccessAnalysis.h | 38 +++-
.../llvm/Analysis/TargetTransformInfoImpl.h | 11 +-
.../llvm/Transforms/Vectorize/SLPVectorizer.h | 1 -
.../Transforms/Vectorize/SLPVectorizer.cpp | 11 +-
.../X86/store-load-forward-conflict.ll | 211 ++++++++++++++++++
5 files changed, 251 insertions(+), 21 deletions(-)
diff --git a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
index dbd0248f2e58f2..d021351f12164a 100644
--- a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
+++ b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
@@ -236,22 +236,36 @@ class MemoryDepChecker {
}
/// Returns true if a memory dependence at byte distance \p Distance between
- /// a store and load (both with element size \p TypeByteSize bytes) would
- /// prevent store-to-load forwarding when the store is widened to
- /// \p VectorStoreSize bytes.
+ /// a store and load would prevent store-to-load forwarding when the store is
+ /// widened to \p VectorStoreSize bytes. \p TypeByteSize is the store element
+ /// size; \p LoadElementSize, when non-zero, is the load's access width.
///
- /// The predicate fires when (a) the load is misaligned w.r.t. the widened
- /// store window (\c Distance is not a multiple of \p VectorStoreSize), and
- /// (b) the conflicting store is still likely to be in the store buffer
- /// (\c Distance / VectorStoreSize is below 8 * TypeByteSize iterations).
- /// Both couldPreventStoreLoadForward and SLPVectorizer use this as their
- /// core STLF cost-model gate.
+ /// The conflicting store must still be likely to be in the store buffer
+ /// (\c Distance / VectorStoreSize is below 8 * TypeByteSize iterations), and
+ /// the load must overlap two widened stores, which happens when either:
+ /// (a) the load is misaligned w.r.t. the widened store window
+ /// (\c Distance is not a multiple of \p VectorStoreSize), or
+ /// (b) the load starts aligned but is itself wider than the widened store
+ /// window (\p LoadElementSize > \p VectorStoreSize), so it overruns
+ /// into the next widened store.
+ /// Both couldPreventStoreLoadForward and SLPVectorizer use this as their core
+ /// STLF cost-model gate. LAA omits \p LoadElementSize (the load and store
+ /// share the element size there, so (b) can never add anything).
static bool isStoreLoadForwardingConflict(uint64_t Distance,
uint64_t VectorStoreSize,
- uint64_t TypeByteSize) {
+ uint64_t TypeByteSize,
+ uint64_t LoadElementSize = 0) {
+ if (VectorStoreSize == 0)
+ return false;
const uint64_t NumItersForStoreLoadThroughMemory = 8 * TypeByteSize;
- return VectorStoreSize != 0 && Distance % VectorStoreSize != 0 &&
- Distance / VectorStoreSize < NumItersForStoreLoadThroughMemory;
+ if (Distance / VectorStoreSize >= NumItersForStoreLoadThroughMemory)
+ return false;
+ // (a) Misaligned load: it straddles two widened stores.
+ if (Distance % VectorStoreSize != 0)
+ return true;
+ // (b) Aligned load that is wider than the widened store still overruns
+ // into the next widened store (LoadElementSize == 0 disables this term).
+ return LoadElementSize > VectorStoreSize;
}
/// Return safe power-of-2 number of elements, which do not prevent store-load
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index f27c20776c24fd..899445e47c48eb 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -23,6 +23,7 @@
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Operator.h"
#include "llvm/IR/PatternMatch.h"
+#include "llvm/MC/MCSchedule.h"
#include <optional>
#include <utility>
@@ -683,11 +684,11 @@ class LLVM_ABI TargetTransformInfoImplBase {
virtual InstructionCost getStoreLoadForwardingConflictCost(
Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
- // No subtarget scheduling model is available here, so fall back to a
- // conservative default STLF stall penalty (~10 cycles). Targets with a real
- // scheduling model go through BasicTTIImpl and use their own value instead.
- constexpr unsigned DefaultStoreLoadForwardingConflictCost = 10;
- return InstructionCost(DefaultStoreLoadForwardingConflictCost);
+ // No subtarget scheduling model is available here, so fall back to the same
+ // default store-to-load forwarding penalty the scheduling model uses.
+ // Targets with a real scheduling model go through BasicTTIImpl and use
+ // their own StoreLoadForwardingPenalty instead.
+ return InstructionCost(MCSchedModel::DefaultStoreLoadForwardingPenalty);
}
virtual std::optional<unsigned>
getCacheSize(TargetTransformInfo::CacheLevel Level) const {
diff --git a/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h b/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h
index b35a49c2274873..538d62626b37b9 100644
--- a/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h
+++ b/llvm/include/llvm/Transforms/Vectorize/SLPVectorizer.h
@@ -19,7 +19,6 @@
#define LLVM_TRANSFORMS_VECTORIZE_SLPVECTORIZER_H
#include "llvm/ADT/ArrayRef.h"
-#include "llvm/ADT/DenseMap.h"
#include "llvm/ADT/MapVector.h"
#include "llvm/ADT/SetVector.h"
#include "llvm/ADT/SmallVector.h"
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 2f1dc8d58c67a1..24f83cc29af5c0 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28367,14 +28367,19 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
LLVM_DEBUG(dbgs() << "SLP: STLF: load=" << *LoadI << " distance="
<< Distance << " bytes from chain base\n");
- // Conflict if the load is misaligned to the wide store within the recency
- // window.
+ // Conflict if the load overlaps two wide stores within the recency window,
+ // either because it is misaligned or because the load itself is wider than
+ // the wide store and overruns its window.
+ TypeSize LoadTypeSize = DL->getTypeStoreSize(LoadI->getType());
+ uint64_t LoadElementSize =
+ LoadTypeSize.isScalable() ? 0 : LoadTypeSize.getFixedValue();
if (MemoryDepChecker::isStoreLoadForwardingConflict(
- Distance, VectorStoreBytes, ElementSize)) {
+ Distance, VectorStoreBytes, ElementSize, LoadElementSize)) {
LLVM_DEBUG(dbgs() << "SLP: Store-load forwarding conflict: "
<< (isVectorized(LoadI) ? "widened" : "scalar")
<< " load, distance " << Distance
<< " bytes, vector store width " << VectorStoreBytes
+ << " bytes, load width " << LoadElementSize
<< " bytes, misalignment "
<< (Distance % VectorStoreBytes) << "\n");
return CacheAndReturn(true);
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index f17d390ee62200..c5aded18fb4622 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -801,3 +801,214 @@ for.body:
for.end:
ret void
}
+
+;
+; Test 12: Aligned load that is WIDER than the widened store (crossing).
+;
+; i8 store chain, VF=4 => widened store window W = 4 bytes. A backward i64
+; load (element size 8 > 4) sits at aligned Distance = 4: 4 % 4 == 0 (not
+; misaligned), but the load itself overruns its window and straddles two
+; widened stores. The misalignment-only test would miss this; the load-width
+; term catches it, so vectorization must be rejected with the check on.
+;
+define void @stlf_conflict_cross_boundary_wider_load(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @stlf_conflict_cross_boundary_wider_load(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[B0:%.*]] = sub i64 [[I]], 4
+; STLF-ON-NEXT: [[G0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[B0]]
+; STLF-ON-NEXT: [[X0:%.*]] = load i64, ptr [[G0]], align 1
+; STLF-ON-NEXT: [[B1:%.*]] = add i64 [[I]], 64
+; STLF-ON-NEXT: [[G1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[B1]]
+; STLF-ON-NEXT: [[X1:%.*]] = load i64, ptr [[G1]], align 1
+; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], 128
+; STLF-ON-NEXT: [[G2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[B2]]
+; STLF-ON-NEXT: [[X2:%.*]] = load i64, ptr [[G2]], align 1
+; STLF-ON-NEXT: [[B3:%.*]] = add i64 [[I]], 192
+; STLF-ON-NEXT: [[G3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[B3]]
+; STLF-ON-NEXT: [[X3:%.*]] = load i64, ptr [[G3]], align 1
+; STLF-ON-NEXT: [[T0:%.*]] = trunc i64 [[X0]] to i8
+; STLF-ON-NEXT: [[T1:%.*]] = trunc i64 [[X1]] to i8
+; STLF-ON-NEXT: [[T2:%.*]] = trunc i64 [[X2]] to i8
+; STLF-ON-NEXT: [[T3:%.*]] = trunc i64 [[X3]] to i8
+; STLF-ON-NEXT: [[S0:%.*]] = add i8 [[T0]], 1
+; STLF-ON-NEXT: [[S1:%.*]] = add i8 [[T1]], 2
+; STLF-ON-NEXT: [[S2:%.*]] = add i8 [[T2]], 3
+; STLF-ON-NEXT: [[S3:%.*]] = add i8 [[T3]], 4
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i8 [[S0]], ptr [[P0]], align 1
+; STLF-ON-NEXT: store i8 [[S1]], ptr [[P1]], align 1
+; STLF-ON-NEXT: store i8 [[S2]], ptr [[P2]], align 1
+; STLF-ON-NEXT: store i8 [[S3]], ptr [[P3]], align 1
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_conflict_cross_boundary_wider_load(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x ptr> poison, ptr [[A]], i64 0
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x ptr> [[TMP0]], <4 x ptr> poison, <4 x i32> zeroinitializer
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP2:%.*]] = insertelement <4 x i64> poison, i64 [[I]], i64 0
+; STLF-OFF-NEXT: [[TMP3:%.*]] = shufflevector <4 x i64> [[TMP2]], <4 x i64> poison, <4 x i32> zeroinitializer
+; STLF-OFF-NEXT: [[TMP4:%.*]] = add <4 x i64> [[TMP3]], <i64 -4, i64 64, i64 128, i64 192>
+; STLF-OFF-NEXT: [[TMP5:%.*]] = getelementptr inbounds i8, <4 x ptr> [[TMP1]], <4 x i64> [[TMP4]]
+; STLF-OFF-NEXT: [[TMP6:%.*]] = call <4 x i64> @llvm.masked.gather.v4i64.v4p0(<4 x ptr> align 1 [[TMP5]], <4 x i1> splat (i1 true), <4 x i64> poison)
+; STLF-OFF-NEXT: [[TMP7:%.*]] = trunc <4 x i64> [[TMP6]] to <4 x i8>
+; STLF-OFF-NEXT: [[TMP8:%.*]] = add <4 x i8> [[TMP7]], <i8 1, i8 2, i8 3, i8 4>
+; STLF-OFF-NEXT: store <4 x i8> [[TMP8]], ptr [[P0]], align 1
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
+
+ %b0 = sub i64 %i, 4
+ %g0 = getelementptr inbounds i8, ptr %A, i64 %b0
+ %x0 = load i64, ptr %g0, align 1
+ %b1 = add i64 %i, 64
+ %g1 = getelementptr inbounds i8, ptr %A, i64 %b1
+ %x1 = load i64, ptr %g1, align 1
+ %b2 = add i64 %i, 128
+ %g2 = getelementptr inbounds i8, ptr %A, i64 %b2
+ %x2 = load i64, ptr %g2, align 1
+ %b3 = add i64 %i, 192
+ %g3 = getelementptr inbounds i8, ptr %A, i64 %b3
+ %x3 = load i64, ptr %g3, align 1
+
+ %t0 = trunc i64 %x0 to i8
+ %t1 = trunc i64 %x1 to i8
+ %t2 = trunc i64 %x2 to i8
+ %t3 = trunc i64 %x3 to i8
+
+ %s0 = add i8 %t0, 1
+ %s1 = add i8 %t1, 2
+ %s2 = add i8 %t2, 3
+ %s3 = add i8 %t3, 4
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+
+ %p0 = getelementptr inbounds i8, ptr %A, i64 %i
+ %p1 = getelementptr inbounds i8, ptr %A, i64 %i1
+ %p2 = getelementptr inbounds i8, ptr %A, i64 %i2
+ %p3 = getelementptr inbounds i8, ptr %A, i64 %i3
+
+ store i8 %s0, ptr %p0, align 1
+ store i8 %s1, ptr %p1, align 1
+ store i8 %s2, ptr %p2, align 1
+ store i8 %s3, ptr %p3, align 1
+
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
+
+;
+; Test 13: Aligned load wider than the element but CONTAINED (no crossing).
+;
+; Same shape as Test 12, but the backward load is i32 (element size 4 == W).
+; Starting aligned, it exactly fills one widened store window and does not
+; overrun into the next one, so it is NOT a forwarding conflict. Vectorization
+; must happen regardless of the STLF check, proving the load-width term does
+; not over-reject aligned, contained loads.
+;
+define void @stlf_no_conflict_wider_contained_load(ptr noalias %A, i64 %n) {
+; CHECK-LABEL: define void @stlf_no_conflict_wider_contained_load(
+; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x ptr> poison, ptr [[A]], i64 0
+; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x ptr> [[TMP0]], <4 x ptr> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT: br label %[[FOR_BODY:.*]]
+; CHECK: [[FOR_BODY]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[TMP2:%.*]] = insertelement <4 x i64> poison, i64 [[I]], i64 0
+; CHECK-NEXT: [[TMP3:%.*]] = shufflevector <4 x i64> [[TMP2]], <4 x i64> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT: [[TMP4:%.*]] = add <4 x i64> [[TMP3]], <i64 -4, i64 64, i64 128, i64 192>
+; CHECK-NEXT: [[TMP5:%.*]] = getelementptr inbounds i8, <4 x ptr> [[TMP1]], <4 x i64> [[TMP4]]
+; CHECK-NEXT: [[TMP6:%.*]] = call <4 x i32> @llvm.masked.gather.v4i32.v4p0(<4 x ptr> align 1 [[TMP5]], <4 x i1> splat (i1 true), <4 x i32> poison)
+; CHECK-NEXT: [[TMP7:%.*]] = trunc <4 x i32> [[TMP6]] to <4 x i8>
+; CHECK-NEXT: [[TMP8:%.*]] = add <4 x i8> [[TMP7]], <i8 1, i8 2, i8 3, i8 4>
+; CHECK-NEXT: store <4 x i8> [[TMP8]], ptr [[P0]], align 1
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; CHECK: [[FOR_END]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
+
+ %b0 = sub i64 %i, 4
+ %g0 = getelementptr inbounds i8, ptr %A, i64 %b0
+ %x0 = load i32, ptr %g0, align 1
+ %b1 = add i64 %i, 64
+ %g1 = getelementptr inbounds i8, ptr %A, i64 %b1
+ %x1 = load i32, ptr %g1, align 1
+ %b2 = add i64 %i, 128
+ %g2 = getelementptr inbounds i8, ptr %A, i64 %b2
+ %x2 = load i32, ptr %g2, align 1
+ %b3 = add i64 %i, 192
+ %g3 = getelementptr inbounds i8, ptr %A, i64 %b3
+ %x3 = load i32, ptr %g3, align 1
+
+ %t0 = trunc i32 %x0 to i8
+ %t1 = trunc i32 %x1 to i8
+ %t2 = trunc i32 %x2 to i8
+ %t3 = trunc i32 %x3 to i8
+
+ %s0 = add i8 %t0, 1
+ %s1 = add i8 %t1, 2
+ %s2 = add i8 %t2, 3
+ %s3 = add i8 %t3, 4
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+
+ %p0 = getelementptr inbounds i8, ptr %A, i64 %i
+ %p1 = getelementptr inbounds i8, ptr %A, i64 %i1
+ %p2 = getelementptr inbounds i8, ptr %A, i64 %i2
+ %p3 = getelementptr inbounds i8, ptr %A, i64 %i3
+
+ store i8 %s0, ptr %p0, align 1
+ store i8 %s1, ptr %p1, align 1
+ store i8 %s2, ptr %p2, align 1
+ store i8 %s3, ptr %p3, align 1
+
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
>From ed29df29bf136ed41cd8e5577074f4690fa6773f Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Thu, 13 Aug 2026 10:41:48 +0530
Subject: [PATCH 04/21] [SLP] Address review: widened load width, same-loop
guard, skip gathered nodes
---
.../llvm/Analysis/TargetTransformInfoImpl.h | 12 +-
llvm/lib/Analysis/TargetTransformInfo.cpp | 10 ++
.../Transforms/Vectorize/SLPVectorizer.cpp | 28 +++-
.../X86/store-load-forward-widened-load.ll | 132 ++++++++++++++++++
4 files changed, 169 insertions(+), 13 deletions(-)
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-widened-load.ll
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 899445e47c48eb..cc8cfd2785aeaf 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -23,7 +23,6 @@
#include "llvm/IR/IntrinsicInst.h"
#include "llvm/IR/Operator.h"
#include "llvm/IR/PatternMatch.h"
-#include "llvm/MC/MCSchedule.h"
#include <optional>
#include <utility>
@@ -682,14 +681,11 @@ class LLVM_ABI TargetTransformInfoImplBase {
virtual unsigned getCacheLineSize() const { return 0; }
+ // Defined out of line in TargetTransformInfo.cpp so this widely-included
+ // header does not need to pull in llvm/MC/MCSchedule.h just to reference the
+ // default store-to-load forwarding penalty.
virtual InstructionCost getStoreLoadForwardingConflictCost(
- Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
- // No subtarget scheduling model is available here, so fall back to the same
- // default store-to-load forwarding penalty the scheduling model uses.
- // Targets with a real scheduling model go through BasicTTIImpl and use
- // their own StoreLoadForwardingPenalty instead.
- return InstructionCost(MCSchedModel::DefaultStoreLoadForwardingPenalty);
- }
+ Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const;
virtual std::optional<unsigned>
getCacheSize(TargetTransformInfo::CacheLevel Level) const {
switch (Level) {
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index b4a57d1c006354..db1c6136551d16 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -20,6 +20,7 @@
#include "llvm/IR/Module.h"
#include "llvm/IR/Operator.h"
#include "llvm/InitializePasses.h"
+#include "llvm/MC/MCSchedule.h"
#include "llvm/Support/CommandLine.h"
#include <optional>
#include <utility>
@@ -1600,6 +1601,15 @@ bool TargetTransformInfo::isUniform(const Instruction *I,
TargetTransformInfoImplBase::~TargetTransformInfoImplBase() = default;
+InstructionCost TargetTransformInfoImplBase::getStoreLoadForwardingConflictCost(
+ Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
+ // No subtarget scheduling model is available here, so fall back to the same
+ // default store-to-load forwarding penalty the scheduling model uses. Targets
+ // with a real scheduling model go through BasicTTIImpl and use their own
+ // StoreLoadForwardingPenalty instead.
+ return InstructionCost(MCSchedModel::DefaultStoreLoadForwardingPenalty);
+}
+
TargetIRAnalysis::TargetIRAnalysis() : TTICallback(&getDefaultTTI) {}
TargetIRAnalysis::TargetIRAnalysis(
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 24f83cc29af5c0..0bf7d9053a8c0c 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28323,7 +28323,8 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
return CacheAndReturn(false);
// Store-to-load forwarding hazards are a loop-carried concern.
- if (!LI->getLoopFor(FirstStore->getParent()))
+ const Loop *StoreL = LI->getLoopFor(FirstStore->getParent());
+ if (!StoreL)
return CacheAndReturn(false);
uint64_t VectorStoreBytes = VF * ElementSize;
@@ -28338,7 +28339,7 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
SmallPtrSet<LoadInst *, 8> CandidateLoads;
for (const std::unique_ptr<TreeEntry> &TEPtr : VectorizableTree) {
const TreeEntry *TE = TEPtr.get();
- if (DeletedNodes.contains(TE))
+ if (DeletedNodes.contains(TE) || TransformedToGatherNodes.contains(TE))
continue;
if (!TE->isGather() &&
!(TE->hasState() && TE->getOpcode() == Instruction::Load))
@@ -28356,6 +28357,9 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
// For each candidate load, the widened chain becomes one wide store at the
// base; check whether the load straddles two such wide stores.
for (LoadInst *LoadI : CandidateLoads) {
+ // Only loads in the store's loop share its loop-carried dependence.
+ if (LI->getLoopFor(LoadI->getParent()) != StoreL)
+ continue;
std::optional<int64_t> Diff =
getPointersDiff(ValueTy, FirstStore->getPointerOperand(),
LoadI->getType(), LoadI->getPointerOperand(), *DL, *SE,
@@ -28367,12 +28371,26 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
LLVM_DEBUG(dbgs() << "SLP: STLF: load=" << *LoadI << " distance="
<< Distance << " bytes from chain base\n");
- // Conflict if the load overlaps two wide stores within the recency window,
- // either because it is misaligned or because the load itself is wider than
- // the wide store and overruns its window.
+ // A widened (regularly vectorized) load accesses the whole vector at once,
+ // so its effective width is VectorFactor * element size, not one element.
+ // Such a wide load can straddle two wide stores even when perfectly
+ // aligned, which the misalignment-only test would miss.
TypeSize LoadTypeSize = DL->getTypeStoreSize(LoadI->getType());
uint64_t LoadElementSize =
LoadTypeSize.isScalable() ? 0 : LoadTypeSize.getFixedValue();
+ for (const TreeEntry *LTE : getTreeEntries(LoadI)) {
+ if (LTE->isGather() || DeletedNodes.contains(LTE) ||
+ TransformedToGatherNodes.contains(LTE))
+ continue;
+ if (LTE->hasState() && LTE->State == TreeEntry::Vectorize &&
+ LTE->getOpcode() == Instruction::Load) {
+ LoadElementSize *= LTE->getVectorFactor();
+ break;
+ }
+ }
+ // Conflict if the load overlaps two wide stores within the recency window,
+ // either because it is misaligned or because the load itself is wider than
+ // the wide store and overruns its window.
if (MemoryDepChecker::isStoreLoadForwardingConflict(
Distance, VectorStoreBytes, ElementSize, LoadElementSize)) {
LLVM_DEBUG(dbgs() << "SLP: Store-load forwarding conflict: "
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-widened-load.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-widened-load.ll
new file mode 100644
index 00000000000000..890b06a477affb
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-widened-load.ll
@@ -0,0 +1,132 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-ON
+; RUN: opt < %s -passes=slp-vectorizer -slp-store-load-forward-check=false -S \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-OFF
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; A widened load can straddle two widened stores even when it is perfectly
+; aligned to the store window, because the load itself is wider than that
+; window. Here four contiguous i32 loads become a 16-byte <4 x i32> load, and
+; the truncated results feed four i8 stores that become a 4-byte <4 x i8>
+; store. The load base is 16 bytes behind the store base:
+; 16 % 4 = 0 -> aligned to the widened store window (misalignment term (a)
+; does not fire), but
+; load width 16 > 4 = widened store width -> the aligned-but-wider term (b)
+; fires: the wide load overruns into the next widened store.
+; 16 / 4 = 4 iterations -> the store is still hot in the store buffer.
+; So SLP must price in the store-to-load forwarding penalty and leave the chain
+; scalar with the check on; with the check off it widens both load and store.
+; This case is reachable only because the STLF gate uses the *widened* load
+; width (VectorFactor * element size) rather than a single element.
+define void @widened_load_crosses(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @widened_load_crosses(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 64, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BM16:%.*]] = add i64 [[I]], -16
+; STLF-ON-NEXT: [[BM12:%.*]] = add i64 [[I]], -12
+; STLF-ON-NEXT: [[BM8:%.*]] = add i64 [[I]], -8
+; STLF-ON-NEXT: [[BM4:%.*]] = add i64 [[I]], -4
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM16]]
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM12]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM8]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM4]]
+; STLF-ON-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[A0:%.*]] = add i32 [[L0]], 1
+; STLF-ON-NEXT: [[A1:%.*]] = add i32 [[L1]], 1
+; STLF-ON-NEXT: [[A2:%.*]] = add i32 [[L2]], 1
+; STLF-ON-NEXT: [[A3:%.*]] = add i32 [[L3]], 1
+; STLF-ON-NEXT: [[T0:%.*]] = trunc i32 [[A0]] to i8
+; STLF-ON-NEXT: [[T1:%.*]] = trunc i32 [[A1]] to i8
+; STLF-ON-NEXT: [[T2:%.*]] = trunc i32 [[A2]] to i8
+; STLF-ON-NEXT: [[T3:%.*]] = trunc i32 [[A3]] to i8
+; STLF-ON-NEXT: [[I1:%.*]] = add i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add i64 [[I]], 3
+; STLF-ON-NEXT: [[S0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[S1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[S2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[S3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i8 [[T0]], ptr [[S0]], align 1
+; STLF-ON-NEXT: store i8 [[T1]], ptr [[S1]], align 1
+; STLF-ON-NEXT: store i8 [[T2]], ptr [[S2]], align 1
+; STLF-ON-NEXT: store i8 [[T3]], ptr [[S3]], align 1
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @widened_load_crosses(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 64, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BM16:%.*]] = add i64 [[I]], -16
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM16]]
+; STLF-OFF-NEXT: [[S0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; STLF-OFF-NEXT: [[TMP1:%.*]] = add <4 x i32> [[TMP0]], splat (i32 1)
+; STLF-OFF-NEXT: [[TMP2:%.*]] = trunc <4 x i32> [[TMP1]] to <4 x i8>
+; STLF-OFF-NEXT: store <4 x i8> [[TMP2]], ptr [[S0]], align 1
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 64, %entry ], [ %i.next, %for.body ]
+ %bm16 = add i64 %i, -16
+ %bm12 = add i64 %i, -12
+ %bm8 = add i64 %i, -8
+ %bm4 = add i64 %i, -4
+ %p0 = getelementptr inbounds i8, ptr %A, i64 %bm16
+ %p1 = getelementptr inbounds i8, ptr %A, i64 %bm12
+ %p2 = getelementptr inbounds i8, ptr %A, i64 %bm8
+ %p3 = getelementptr inbounds i8, ptr %A, i64 %bm4
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+ %a0 = add i32 %l0, 1
+ %a1 = add i32 %l1, 1
+ %a2 = add i32 %l2, 1
+ %a3 = add i32 %l3, 1
+ %t0 = trunc i32 %a0 to i8
+ %t1 = trunc i32 %a1 to i8
+ %t2 = trunc i32 %a2 to i8
+ %t3 = trunc i32 %a3 to i8
+ %i1 = add i64 %i, 1
+ %i2 = add i64 %i, 2
+ %i3 = add i64 %i, 3
+ %s0 = getelementptr inbounds i8, ptr %A, i64 %i
+ %s1 = getelementptr inbounds i8, ptr %A, i64 %i1
+ %s2 = getelementptr inbounds i8, ptr %A, i64 %i2
+ %s3 = getelementptr inbounds i8, ptr %A, i64 %i3
+ store i8 %t0, ptr %s0, align 1
+ store i8 %t1, ptr %s1, align 1
+ store i8 %t2, ptr %s2, align 1
+ store i8 %t3, ptr %s3, align 1
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK: {{.*}}
>From 5799499ff50fbc45c1d6e299cf0dd92c70015608 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Tue, 18 Aug 2026 07:09:19 +0530
Subject: [PATCH 05/21] [SLP] Make CacheAndReturn result parameter const
(review nit)
---
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 0bf7d9053a8c0c..61fd386fac397f 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28298,7 +28298,7 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
if (CacheIt != StlfConflictCache.end())
return CacheIt->second;
- auto CacheAndReturn = [&](bool Result) -> bool {
+ auto CacheAndReturn = [&](const bool Result) -> bool {
StlfConflictCache[Key] = Result;
// No conflict at VF implies none at any smaller VF that divides VF, so seed
// those entries too. The recency window is VF-independent (so it stays
>From b6028674595ebe76bb6006f34d73638bf22e03a7 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Thu, 20 Aug 2026 11:32:54 +0530
Subject: [PATCH 06/21] [SLP][NFC] Use ASCII in store-load-forward-conflict
test comments
Replace Unicode arrows/em-dashes with ASCII (-> and --) in the descriptive comments of store-load-forward-conflict.ll. Comment-only; no functional or CHECK-line change.
---
.../SLPVectorizer/X86/store-load-forward-conflict.ll | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index c5aded18fb4622..87fd36628b45d5 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -24,7 +24,7 @@ target triple = "x86_64-unknown-linux-gnu"
; 20 % 16 = 4 -> misaligned, load straddles two pending vector stores
; 20 / 16 = 1 -> store still hot in store buffer
; The STLF penalty makes the VF=4 store entry unprofitable. Sub-chains at
-; VF=2 are priced per-base: {A[i+1],A[i+2]} has distance 24, 24%8=0 → no
+; VF=2 are priced per-base: {A[i+1],A[i+2]} has distance 24, 24%8=0 -> no
; conflict, no penalty, so it still vectorizes.
;
define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
@@ -560,10 +560,10 @@ for.end:
}
;
-; Test 9: Motivating example — a[i] = a[i-1] + 1.
+; Test 9: Motivating example -- a[i] = a[i-1] + 1.
;
; Distance = 4 bytes (1 element) from chain base. VectorStoreBytes = 16
-; at VF=4. 4 % 16 = 4 → misaligned → STLF conflict. The full 4-wide store
+; at VF=4. 4 % 16 = 4 -> misaligned -> STLF conflict. The full 4-wide store
; entry is penalized and is not profitable. Sub-chains that are individually
; misaligned are penalized too; only sub-chains where the distance happens to
; be aligned to the narrower vector store width vectorize.
>From 576b8251a26243476cb016f6ee8a3be194f77883 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Thu, 20 Aug 2026 11:32:54 +0530
Subject: [PATCH 07/21] [SLP] Enumerate all loop loads for STLF and drop the
conflict cache
- Enumerate STLF candidate loads from every simple load in the store's loop instead of only the SLP tree's load/gather nodes. A conflicting load may feed a scalar store or be vectorized in a different tree and still break store-to-load forwarding for the widened store, so it need not appear as a load node of the chain being costed.
- Remove StlfConflictCache. Instrumented across the full SLP test corpus it never hits: deleteTree() clears it on every buildTree() and each (store, VF) pair is queried once per attempt, so no entry is ever reused.
- Keep the fallback store-to-load-forwarding penalty in the TTI layer as its own constexpr so Analysis no longer includes llvm/MC/MCSchedule.h, dropping the Analysis->MC dependency (value stays 10; real targets go through BasicTTIImpl).
- Regenerate affected checks: Test 7 now stays scalar under STLF-ON, and memory-runtime-checks.ll.
---
.../llvm/Analysis/TargetTransformInfoImpl.h | 16 +++-
llvm/lib/Analysis/TargetTransformInfo.cpp | 10 --
.../Transforms/Vectorize/SLPVectorizer.cpp | 67 ++++----------
.../X86/memory-runtime-checks.ll | 20 ++--
.../X86/store-load-forward-conflict.ll | 91 +++++++++++++------
5 files changed, 103 insertions(+), 101 deletions(-)
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index cc8cfd2785aeaf..c95e8b0844ebd2 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -681,11 +681,19 @@ class LLVM_ABI TargetTransformInfoImplBase {
virtual unsigned getCacheLineSize() const { return 0; }
- // Defined out of line in TargetTransformInfo.cpp so this widely-included
- // header does not need to pull in llvm/MC/MCSchedule.h just to reference the
- // default store-to-load forwarding penalty.
+ /// Default store-to-load forwarding stall penalty (in cycles) charged when no
+ /// subtarget scheduling model is available. Kept intentionally in sync with
+ /// MCSchedModel::DefaultStoreLoadForwardingPenalty (the scheduling-model field
+ /// default); the cost layer keeps its own copy so it does not depend on MC.
+ static constexpr unsigned DefaultStoreLoadForwardingPenalty = 10;
+
virtual InstructionCost getStoreLoadForwardingConflictCost(
- Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const;
+ Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
+ // No subtarget scheduling model is available here, so fall back to the
+ // default penalty. Targets with a real scheduling model go through
+ // BasicTTIImpl and use their own StoreLoadForwardingPenalty instead.
+ return InstructionCost(DefaultStoreLoadForwardingPenalty);
+ }
virtual std::optional<unsigned>
getCacheSize(TargetTransformInfo::CacheLevel Level) const {
switch (Level) {
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index db1c6136551d16..b4a57d1c006354 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -20,7 +20,6 @@
#include "llvm/IR/Module.h"
#include "llvm/IR/Operator.h"
#include "llvm/InitializePasses.h"
-#include "llvm/MC/MCSchedule.h"
#include "llvm/Support/CommandLine.h"
#include <optional>
#include <utility>
@@ -1601,15 +1600,6 @@ bool TargetTransformInfo::isUniform(const Instruction *I,
TargetTransformInfoImplBase::~TargetTransformInfoImplBase() = default;
-InstructionCost TargetTransformInfoImplBase::getStoreLoadForwardingConflictCost(
- Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
- // No subtarget scheduling model is available here, so fall back to the same
- // default store-to-load forwarding penalty the scheduling model uses. Targets
- // with a real scheduling model go through BasicTTIImpl and use their own
- // StoreLoadForwardingPenalty instead.
- return InstructionCost(MCSchedModel::DefaultStoreLoadForwardingPenalty);
-}
-
TargetIRAnalysis::TargetIRAnalysis() : TTICallback(&getDefaultTTI) {}
TargetIRAnalysis::TargetIRAnalysis(
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 61fd386fac397f..111f1846cca458 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -1013,8 +1013,7 @@ class slpvectorizer::BoUpSLP {
/// \returns true if widening the store chain anchored at \p BaseStore into a
/// vector store of \p VF elements would break store-to-load forwarding for a
/// nearby loop-carried load (a short, misaligned backward dependence). Used
- /// to add an STLF penalty to the store entry's cost. Result is memoized in
- /// StlfConflictCache.
+ /// to add an STLF penalty to the store entry's cost.
bool findStoreLoadForwardingConflict(StoreInst *BaseStore, unsigned VF);
/// Calculates the cost of the subtrees, trims non-profitable ones and returns
@@ -1252,9 +1251,6 @@ class slpvectorizer::BoUpSLP {
TreeEntryToStridedPtrInfoMap.clear();
CurrentLoopNest.clear();
MergedLoopBTCs.clear();
- // Loads/stores may be reconstructed on the next vectorization attempt, so
- // stale STLF decisions must not carry over.
- StlfConflictCache.clear();
}
unsigned getTreeSize() const { return VectorizableTree.size(); }
@@ -5700,11 +5696,6 @@ class slpvectorizer::BoUpSLP {
const DataLayout *DL;
OptimizationRemarkEmitter *ORE;
- /// Cached STLF conflict decisions keyed by (base store of chain, VF), to
- /// avoid re-walking the tree when the store entry is costed repeatedly.
- /// Cleared on each buildTree() via deleteTree().
- SmallDenseMap<std::pair<const StoreInst *, unsigned>, bool> StlfConflictCache;
-
unsigned MaxVecRegSize; // This is set by TTI or overridden by cl::opt.
unsigned MinVecRegSize; // Set by cl::opt (default: 128).
@@ -28292,67 +28283,41 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
StoreInst *FirstStore = BaseStore;
- // Memoize per (store, VF); the entry is re-costed repeatedly.
- auto Key = std::make_pair(FirstStore, VF);
- auto CacheIt = StlfConflictCache.find(Key);
- if (CacheIt != StlfConflictCache.end())
- return CacheIt->second;
-
- auto CacheAndReturn = [&](const bool Result) -> bool {
- StlfConflictCache[Key] = Result;
- // No conflict at VF implies none at any smaller VF that divides VF, so seed
- // those entries too. The recency window is VF-independent (so it stays
- // satisfied for any smaller width), and the distance remains a multiple of
- // V * ElementSize only for divisors of VF. This covers the power-of-2
- // ladder (each smaller power of 2 divides the larger) and any
- // non-power-of-2 VF the caller may probe. Conflicts do not propagate
- // downward.
- if (!Result)
- for (unsigned V = 2; V < VF; ++V)
- if (VF % V == 0)
- StlfConflictCache.try_emplace(std::make_pair(FirstStore, V), false);
- return Result;
- };
-
Type *ValueTy = FirstStore->getValueOperand()->getType();
TypeSize StoreSize = DL->getTypeStoreSize(ValueTy);
if (StoreSize.isScalable())
- return CacheAndReturn(false);
+ return false;
uint64_t ElementSize = StoreSize.getFixedValue();
if (ElementSize == 0)
- return CacheAndReturn(false);
+ return false;
// Store-to-load forwarding hazards are a loop-carried concern.
const Loop *StoreL = LI->getLoopFor(FirstStore->getParent());
if (!StoreL)
- return CacheAndReturn(false);
+ return false;
uint64_t VectorStoreBytes = VF * ElementSize;
LLVM_DEBUG(dbgs() << "SLP: STLF check: VF=" << VF
<< " ElementSize=" << ElementSize
<< " VectorStoreBytes=" << VectorStoreBytes << "\n");
- // Enumerate candidate loads directly from the tree's load and gather nodes: a
- // conflicting load is either widened (a load node) or packed into a gather
- // leaf (e.g. a splat), so scanning those node kinds is sufficient.
+ // A store-to-load forwarding hazard can involve any load in the loop that
+ // reads the widened store's base, not only loads that became SLP tree nodes.
+ // A conflicting load may feed a scalar store, sit below a gather/splat leaf,
+ // or be vectorized in a different tree, so it need not appear as a load node
+ // of the chain being costed. Enumerate every simple load in the store's loop
+ // that shares the store base.
Value *StoreBase = getUnderlyingObject(FirstStore->getPointerOperand());
SmallPtrSet<LoadInst *, 8> CandidateLoads;
- for (const std::unique_ptr<TreeEntry> &TEPtr : VectorizableTree) {
- const TreeEntry *TE = TEPtr.get();
- if (DeletedNodes.contains(TE) || TransformedToGatherNodes.contains(TE))
- continue;
- if (!TE->isGather() &&
- !(TE->hasState() && TE->getOpcode() == Instruction::Load))
- continue;
- for (Value *V : TE->Scalars)
- if (auto *LoadI = dyn_cast<LoadInst>(V))
+ for (BasicBlock *BB : StoreL->blocks())
+ for (Instruction &I : *BB)
+ if (auto *LoadI = dyn_cast<LoadInst>(&I))
if (LoadI->isSimple() &&
getUnderlyingObject(LoadI->getPointerOperand()) == StoreBase)
CandidateLoads.insert(LoadI);
- }
if (CandidateLoads.empty())
- return CacheAndReturn(false);
+ return false;
// For each candidate load, the widened chain becomes one wide store at the
// base; check whether the load straddles two such wide stores.
@@ -28400,11 +28365,11 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
<< " bytes, load width " << LoadElementSize
<< " bytes, misalignment "
<< (Distance % VectorStoreBytes) << "\n");
- return CacheAndReturn(true);
+ return true;
}
}
- return CacheAndReturn(false);
+ return false;
}
std::optional<bool>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
index 2d19a8c22cac18..ccd69559b093f9 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
@@ -213,27 +213,33 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
; CHECK-NEXT: [[TMP26:%.*]] = getelementptr float, ptr [[ARG4]], i64 6
; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP24]], align 4
; CHECK-NEXT: [[TMP27:%.*]] = load float, ptr [[ARG5:%.*]], align 4
+; CHECK-NEXT: [[TMP28:%.*]] = getelementptr float, ptr [[ARG4]], i64 5
; CHECK-NEXT: [[TMP29:%.*]] = fadd float 0.000000e+00, 0.000000e+00
; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP26]], align 4
; CHECK-NEXT: [[TMP30:%.*]] = getelementptr float, ptr [[ARG4]], i64 4
+; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP28]], align 4
; CHECK-NEXT: [[TMP31:%.*]] = fadd float 0.000000e+00, 0.000000e+00
-; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP30]], align 4
+; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP30]], align 4
; CHECK-NEXT: br label [[BB33:%.*]]
; CHECK: bb33:
; CHECK-NEXT: br label [[BB34:%.*]]
; CHECK: bb34:
; CHECK-NEXT: [[TMP35:%.*]] = getelementptr float, ptr [[ARG4]], i64 3
+; CHECK-NEXT: [[ARG3:%.*]] = getelementptr float, ptr [[ARG4]], i64 2
; CHECK-NEXT: [[TMP37:%.*]] = load float, ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP38:%.*]] = fadd float 0.000000e+00, [[TMP37]]
; CHECK-NEXT: store float [[TMP38]], ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP39:%.*]] = getelementptr float, ptr [[ARG4]], i64 1
-; CHECK-NEXT: [[TMP4:%.*]] = load <2 x float>, ptr [[TMP39]], align 4
-; CHECK-NEXT: [[TMP5:%.*]] = fadd <2 x float> zeroinitializer, [[TMP4]]
-; CHECK-NEXT: store <2 x float> [[TMP5]], ptr [[TMP39]], align 4
-; CHECK-NEXT: [[TMP44:%.*]] = load float, ptr [[ARG3:%.*]], align 4
-; CHECK-NEXT: [[TMP45:%.*]] = load float, ptr [[ARG4]], align 4
+; CHECK-NEXT: [[TMP44:%.*]] = load float, ptr [[ARG3]], align 4
+; CHECK-NEXT: [[TMP41:%.*]] = fadd float 0.000000e+00, [[TMP44]]
+; CHECK-NEXT: store float [[TMP41]], ptr [[ARG3]], align 4
+; CHECK-NEXT: [[TMP45:%.*]] = load float, ptr [[TMP39]], align 4
; CHECK-NEXT: [[TMP46:%.*]] = fadd float 0.000000e+00, [[TMP45]]
-; CHECK-NEXT: store float [[TMP46]], ptr [[ARG4]], align 4
+; CHECK-NEXT: store float [[TMP46]], ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP47:%.*]] = load float, ptr [[ARG6:%.*]], align 4
+; CHECK-NEXT: [[TMP48:%.*]] = load float, ptr [[ARG4]], align 4
+; CHECK-NEXT: [[TMP49:%.*]] = fadd float 0.000000e+00, [[TMP48]]
+; CHECK-NEXT: store float [[TMP49]], ptr [[ARG4]], align 4
; CHECK-NEXT: call void @quux()
; CHECK-NEXT: br label [[BB47:%.*]]
; CHECK: bb47:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index 87fd36628b45d5..398b70586d3d59 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -417,37 +417,70 @@ for.end:
;
; Test 7: Loads that feed scalar arithmetic below a gather/splat leaf.
;
-; A[i-5] and A[i-7] would be misaligned to a 16-byte vector store, but they
-; only feed the scalar %sum, which is then broadcast. Candidate loads are
-; enumerated from the tree's load and gather nodes, so loads sitting below a
-; gather leaf (here, under %sum) are not seen and no penalty is charged; the
-; chain vectorizes. This documents the reachability limit of the node-based
-; enumeration.
+; A[i-5] and A[i-7] are misaligned to a 16-byte vector store (distances 20 and
+; 28 bytes, both %16 != 0) but only feed the scalar %sum, which is broadcast, so
+; they never become load tree nodes of the chain. Candidate loads are now
+; enumerated from all simple loads in the store's loop, not just tree nodes, so
+; these loads are seen and the chain is kept scalar under STLF-ON.
;
define void @stlf_multiple_conflicting_loads(ptr noalias %A, i64 %n) {
-; CHECK-LABEL: define void @stlf_multiple_conflicting_loads(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; CHECK-NEXT: [[ENTRY:.*]]:
-; CHECK-NEXT: br label %[[FOR_BODY:.*]]
-; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
-; CHECK-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
-; CHECK-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
-; CHECK-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
-; CHECK-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
-; CHECK-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
-; CHECK-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
-; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[SUM]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; CHECK: [[FOR_END]]:
-; CHECK-NEXT: ret void
+; STLF-ON-LABEL: define void @stlf_multiple_conflicting_loads(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-ON-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
+; STLF-ON-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
+; STLF-ON-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
+; STLF-ON-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
+; STLF-ON-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
+; STLF-ON-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[SUM]], 1
+; STLF-ON-NEXT: [[T2:%.*]] = add nsw i32 [[SUM]], 2
+; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[SUM]], 3
+; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[SUM]], 4
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: store i32 [[T2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP2]], align 4
+; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_multiple_conflicting_loads(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
+; STLF-OFF-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
+; STLF-OFF-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
+; STLF-OFF-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
+; STLF-OFF-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
+; STLF-OFF-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
+; STLF-OFF-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[SUM]], i64 0
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
;
entry:
br label %for.body
>From 1f6a29e67e727e98e4b58e09445bbf369c764167 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Thu, 20 Aug 2026 18:45:24 +0530
Subject: [PATCH 08/21] [SLP] Make store-to-load forwarding cost stride-aware
The SLP store-to-load forwarding (STLF) check reuses the stride-blind
MemoryDepChecker::isStoreLoadForwardingConflict predicate, which flags a
widened store whenever a nearby backward load is misaligned or wider than
the store window. Because it ignores the loop-carried stride, it also
penalizes strided-independent load/store pairs whose bytes are never
re-read by a future iteration (e.g. the neoverse-v2 interleave_vec O3
case, where a 24-byte stride steps the load window clean past the store).
Gate the penalty on an actual loop-carried re-read: with a common positive
byte stride S equal for the load and the store, the store's bytes are
re-read iff there is an integer k >= 1 with
Distance - LoadElementSize < k * S < Distance + VectorStoreBytes.
If no such k exists the accesses are strided-independent and there is no
forwarding hazard. The stride is read directly from the pointer's affine
SCEV recurrence (the same notion as LoopAccessAnalysis's CommonStride)
rather than via getPtrStride, whose no-wrap versioning and
PredicatedScalarEvolution machinery is meant for legality, not costing.
When the stride is unknown, non-positive, or differs between the load and
the store, fall back to the existing conservative check so no real hazard
is missed.
Add a strided-independent test (with a strided-dependent negative control)
and regenerate the existing STLF golden tests, which now vectorize the
safe sub-chains they previously left scalar.
---
.../llvm/Analysis/TargetTransformInfoImpl.h | 5 +-
.../Transforms/Vectorize/SLPVectorizer.cpp | 50 +++++
.../X86/store-load-forward-conflict.ll | 20 +-
.../X86/store-load-forward-cost.ll | 10 +-
.../X86/store-load-forward-non-power-of-2.ll | 10 +-
.../X86/store-load-forward-profitable.ll | 20 +-
.../store-load-forward-strided-independent.ll | 194 ++++++++++++++++++
7 files changed, 277 insertions(+), 32 deletions(-)
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-strided-independent.ll
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index c95e8b0844ebd2..427fd5a93a6c4d 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -683,8 +683,9 @@ class LLVM_ABI TargetTransformInfoImplBase {
/// Default store-to-load forwarding stall penalty (in cycles) charged when no
/// subtarget scheduling model is available. Kept intentionally in sync with
- /// MCSchedModel::DefaultStoreLoadForwardingPenalty (the scheduling-model field
- /// default); the cost layer keeps its own copy so it does not depend on MC.
+ /// MCSchedModel::DefaultStoreLoadForwardingPenalty (the scheduling-model
+ /// field default); the cost layer keeps its own copy so it does not depend on
+ /// MC.
static constexpr unsigned DefaultStoreLoadForwardingPenalty = 10;
virtual InstructionCost getStoreLoadForwardingConflictCost(
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 111f1846cca458..2f85064f36966e 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28276,6 +28276,26 @@ bool SLPVectorizerPass::runImpl(Function &F, ScalarEvolution *SE_,
return Changed;
}
+/// Returns the constant loop-carried byte stride of \p Ptr in \p L, i.e. the
+/// step of its affine SCEV recurrence, or std::nullopt when \p Ptr is not a
+/// simple affine recurrence in \p L with a constant step. This is the same
+/// notion as LoopAccessAnalysis's CommonStride; it is computed here directly
+/// from ScalarEvolution rather than via getPtrStride, whose no-wrap versioning
+/// and PredicatedScalarEvolution machinery is meant for legality, not costing.
+static std::optional<int64_t>
+getConstantLoopStrideInBytes(Value *Ptr, ScalarEvolution &SE, const Loop *L) {
+ const auto *AR = dyn_cast<SCEVAddRecExpr>(SE.getSCEV(Ptr));
+ if (!AR || AR->getLoop() != L)
+ return std::nullopt;
+ const auto *Step = dyn_cast<SCEVConstant>(AR->getStepRecurrence(SE));
+ if (!Step)
+ return std::nullopt;
+ const APInt &StepVal = Step->getAPInt();
+ if (StepVal.getSignificantBits() > 64)
+ return std::nullopt;
+ return StepVal.getSExtValue();
+}
+
bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
unsigned VF) {
if (!BaseStore)
@@ -28301,6 +28321,12 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
<< " ElementSize=" << ElementSize
<< " VectorStoreBytes=" << VectorStoreBytes << "\n");
+ // Loop-carried byte stride of the store. A conflict is only a real hazard if
+ // a future iteration's load re-reads the bytes this store wrote, which is a
+ // property of the stride (see the per-load check below).
+ std::optional<int64_t> StoreStride = getConstantLoopStrideInBytes(
+ FirstStore->getPointerOperand(), *SE, StoreL);
+
// A store-to-load forwarding hazard can involve any load in the loop that
// reads the widened store's base, not only loads that became SLP tree nodes.
// A conflicting load may feed a scalar store, sit below a gather/splat leaf,
@@ -28353,6 +28379,30 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
break;
}
}
+ // A conflict is only a real hazard if a future iteration's load actually
+ // re-reads the bytes this store wrote. With a common positive loop-carried
+ // stride S, equal for the load and the store, the store's bytes are re-read
+ // iff there is an integer k >= 1 with
+ // Distance - LoadElementSize < k * S < Distance + VectorStoreBytes.
+ // If no such k exists the accesses are strided-independent, so there is no
+ // forwarding hazard. When the stride is unknown, non-positive, or differs
+ // between load and store, fall back to the conservative check below.
+ std::optional<int64_t> LoadStride =
+ getConstantLoopStrideInBytes(LoadI->getPointerOperand(), *SE, StoreL);
+ if (StoreStride && LoadStride && *StoreStride == *LoadStride &&
+ *StoreStride > 0) {
+ int64_t Stride = *StoreStride;
+ int64_t Lo = static_cast<int64_t>(Distance) -
+ static_cast<int64_t>(LoadElementSize);
+ int64_t Hi = static_cast<int64_t>(Distance) +
+ static_cast<int64_t>(VectorStoreBytes);
+ int64_t K = Lo < Stride ? 1 : (Lo / Stride) + 1;
+ if (Stride * K >= Hi) {
+ LLVM_DEBUG(dbgs() << "SLP: STLF: strided-independent (stride " << Stride
+ << "), no future re-read -> no conflict\n");
+ continue;
+ }
+ }
// Conflict if the load overlaps two wide stores within the recency window,
// either because it is misaligned or because the load itself is wider than
// the wide store and overruns its window.
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index 398b70586d3d59..6ac6afbff18cf6 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -37,18 +37,18 @@ define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
+; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
-; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
+; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
@@ -611,18 +611,18 @@ define void @stlf_conflict_short_backward(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
+; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
-; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
+; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
index d371948c3eb6b7..56210cfca11b15 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
@@ -28,18 +28,18 @@ define void @stlf_recurrence(ptr noalias %A, i64 %n) {
; PENALTY-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
; PENALTY-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
; PENALTY-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; PENALTY-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
+; PENALTY-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
; PENALTY-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
-; PENALTY-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; PENALTY-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
; PENALTY-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; PENALTY-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; PENALTY-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; PENALTY-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; PENALTY-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
; PENALTY-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
; PENALTY-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; PENALTY-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
-; PENALTY-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; PENALTY-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
+; PENALTY-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; PENALTY-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
; PENALTY-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
; PENALTY-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; PENALTY-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
index 74433ed96fdd84..4a1c6ef148f133 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
@@ -38,17 +38,17 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[T]], 1
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 2, i32 3>
-; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
+; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
; STLF-ON-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
; STLF-ON-NEXT: [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> poison, <4 x i32> zeroinitializer
; STLF-ON-NEXT: [[TMP5:%.*]] = add nsw <4 x i32> [[TMP4]], <i32 4, i32 5, i32 6, i32 7>
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
index 4b22025a1f4bd1..211848ed84a68b 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
@@ -59,26 +59,26 @@ define void @stlf_profitable(ptr noalias %A, i64 %n) {
; VETO-NEXT: [[T:%.*]] = load i32, ptr [[BGEP]], align 4
; VETO-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
; VETO-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; VETO-NEXT: [[TMP2:%.*]] = add <2 x i32> [[TMP1]], <i32 1, i32 4>
+; VETO-NEXT: [[TMP2:%.*]] = add <2 x i32> [[TMP1]], <i32 3, i32 4>
; VETO-NEXT: [[TMP3:%.*]] = mul <2 x i32> [[TMP2]], [[TMP2]]
-; VETO-NEXT: [[TMP4:%.*]] = add <2 x i32> [[TMP3]], <i32 7, i32 10>
+; VETO-NEXT: [[TMP4:%.*]] = add <2 x i32> [[TMP3]], <i32 9, i32 10>
; VETO-NEXT: [[TMP5:%.*]] = mul <2 x i32> [[TMP4]], [[TMP2]]
-; VETO-NEXT: [[TMP6:%.*]] = add <2 x i32> [[TMP5]], <i32 11, i32 14>
+; VETO-NEXT: [[TMP6:%.*]] = add <2 x i32> [[TMP5]], <i32 13, i32 14>
; VETO-NEXT: [[TMP7:%.*]] = mul <2 x i32> [[TMP6]], [[TMP4]]
-; VETO-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; VETO-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
; VETO-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; VETO-NEXT: [[G0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; VETO-NEXT: [[G1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; VETO-NEXT: [[G3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; VETO-NEXT: [[TMP8:%.*]] = extractelement <2 x i32> [[TMP7]], i64 0
-; VETO-NEXT: store i32 [[TMP8]], ptr [[G0]], align 4
-; VETO-NEXT: [[TMP9:%.*]] = add <2 x i32> [[TMP1]], <i32 2, i32 3>
+; VETO-NEXT: [[TMP9:%.*]] = add <2 x i32> [[TMP1]], <i32 1, i32 2>
; VETO-NEXT: [[TMP10:%.*]] = mul <2 x i32> [[TMP9]], [[TMP9]]
-; VETO-NEXT: [[TMP11:%.*]] = add <2 x i32> [[TMP10]], <i32 8, i32 9>
+; VETO-NEXT: [[TMP11:%.*]] = add <2 x i32> [[TMP10]], <i32 7, i32 8>
; VETO-NEXT: [[TMP12:%.*]] = mul <2 x i32> [[TMP11]], [[TMP9]]
-; VETO-NEXT: [[TMP13:%.*]] = add <2 x i32> [[TMP12]], <i32 12, i32 13>
+; VETO-NEXT: [[TMP13:%.*]] = add <2 x i32> [[TMP12]], <i32 11, i32 12>
; VETO-NEXT: [[TMP14:%.*]] = mul <2 x i32> [[TMP13]], [[TMP11]]
-; VETO-NEXT: store <2 x i32> [[TMP14]], ptr [[G1]], align 4
+; VETO-NEXT: store <2 x i32> [[TMP14]], ptr [[G0]], align 4
+; VETO-NEXT: [[TMP16:%.*]] = extractelement <2 x i32> [[TMP7]], i64 0
+; VETO-NEXT: store i32 [[TMP16]], ptr [[G1]], align 4
; VETO-NEXT: [[TMP15:%.*]] = extractelement <2 x i32> [[TMP7]], i64 1
; VETO-NEXT: store i32 [[TMP15]], ptr [[G3]], align 4
; VETO-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-strided-independent.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-strided-independent.ll
new file mode 100644
index 00000000000000..b3d3bb5841f710
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-strided-independent.ll
@@ -0,0 +1,194 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-ON
+; RUN: opt < %s -passes=slp-vectorizer -slp-store-load-forward-check=false -S \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-OFF
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Both functions have the same widened-load shape as
+; store-load-forward-widened-load.ll: a 16-byte <4 x i32> load feeding a 4-byte
+; <4 x i8> store, with the load base 16 bytes behind the store base
+; (D=16, L=16, V=4). They differ only in the loop-carried stride, which is what
+; decides whether the store's bytes are ever re-read:
+; real hazard <=> exists k>=1 with D - L < k*Stride < D + V.
+
+; Stride 20: 0 < 20*k < 20 has no solution, so no future iteration re-reads the
+; store's bytes. The accesses are strided-independent: there is no forwarding
+; hazard and SLP must widen the store even with the STLF check on.
+define void @strided_independent_wide_load(ptr noalias %A, i64 %n) {
+; CHECK-LABEL: define void @strided_independent_wide_load(
+; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: br label %[[FOR_BODY:.*]]
+; CHECK: [[FOR_BODY]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 64, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[BM16:%.*]] = add i64 [[I]], -16
+; CHECK-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM16]]
+; CHECK-NEXT: [[S0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; CHECK-NEXT: [[TMP1:%.*]] = add <4 x i32> [[TMP0]], splat (i32 1)
+; CHECK-NEXT: [[TMP2:%.*]] = trunc <4 x i32> [[TMP1]] to <4 x i8>
+; CHECK-NEXT: store <4 x i8> [[TMP2]], ptr [[S0]], align 1
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 20
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; CHECK: [[FOR_END]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 64, %entry ], [ %i.next, %for.body ]
+ %bm16 = add i64 %i, -16
+ %bm12 = add i64 %i, -12
+ %bm8 = add i64 %i, -8
+ %bm4 = add i64 %i, -4
+ %p0 = getelementptr inbounds i8, ptr %A, i64 %bm16
+ %p1 = getelementptr inbounds i8, ptr %A, i64 %bm12
+ %p2 = getelementptr inbounds i8, ptr %A, i64 %bm8
+ %p3 = getelementptr inbounds i8, ptr %A, i64 %bm4
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+ %a0 = add i32 %l0, 1
+ %a1 = add i32 %l1, 1
+ %a2 = add i32 %l2, 1
+ %a3 = add i32 %l3, 1
+ %t0 = trunc i32 %a0 to i8
+ %t1 = trunc i32 %a1 to i8
+ %t2 = trunc i32 %a2 to i8
+ %t3 = trunc i32 %a3 to i8
+ %i1 = add i64 %i, 1
+ %i2 = add i64 %i, 2
+ %i3 = add i64 %i, 3
+ %s0 = getelementptr inbounds i8, ptr %A, i64 %i
+ %s1 = getelementptr inbounds i8, ptr %A, i64 %i1
+ %s2 = getelementptr inbounds i8, ptr %A, i64 %i2
+ %s3 = getelementptr inbounds i8, ptr %A, i64 %i3
+ store i8 %t0, ptr %s0, align 1
+ store i8 %t1, ptr %s1, align 1
+ store i8 %t2, ptr %s2, align 1
+ store i8 %t3, ptr %s3, align 1
+ %i.next = add nuw nsw i64 %i, 20
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
+
+; Negative control: identical shape but stride 4. Now 0 < 4*k < 20 holds for
+; k=1..4, so four iterations later the wide load re-reads these exact bytes -
+; a real store-to-load forwarding hazard. SLP must keep the store scalar with
+; the check on, and only widens it when the check is disabled.
+define void @strided_dependent_wide_load(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @strided_dependent_wide_load(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 64, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BM16:%.*]] = add i64 [[I]], -16
+; STLF-ON-NEXT: [[BM12:%.*]] = add i64 [[I]], -12
+; STLF-ON-NEXT: [[BM8:%.*]] = add i64 [[I]], -8
+; STLF-ON-NEXT: [[BM4:%.*]] = add i64 [[I]], -4
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM16]]
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM12]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM8]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM4]]
+; STLF-ON-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[A0:%.*]] = add i32 [[L0]], 1
+; STLF-ON-NEXT: [[A1:%.*]] = add i32 [[L1]], 1
+; STLF-ON-NEXT: [[A2:%.*]] = add i32 [[L2]], 1
+; STLF-ON-NEXT: [[A3:%.*]] = add i32 [[L3]], 1
+; STLF-ON-NEXT: [[T0:%.*]] = trunc i32 [[A0]] to i8
+; STLF-ON-NEXT: [[T1:%.*]] = trunc i32 [[A1]] to i8
+; STLF-ON-NEXT: [[T2:%.*]] = trunc i32 [[A2]] to i8
+; STLF-ON-NEXT: [[T3:%.*]] = trunc i32 [[A3]] to i8
+; STLF-ON-NEXT: [[I1:%.*]] = add i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add i64 [[I]], 3
+; STLF-ON-NEXT: [[S0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[S1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[S2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[S3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i8 [[T0]], ptr [[S0]], align 1
+; STLF-ON-NEXT: store i8 [[T1]], ptr [[S1]], align 1
+; STLF-ON-NEXT: store i8 [[T2]], ptr [[S2]], align 1
+; STLF-ON-NEXT: store i8 [[T3]], ptr [[S3]], align 1
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @strided_dependent_wide_load(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 64, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BM16:%.*]] = add i64 [[I]], -16
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM16]]
+; STLF-OFF-NEXT: [[S0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; STLF-OFF-NEXT: [[TMP1:%.*]] = add <4 x i32> [[TMP0]], splat (i32 1)
+; STLF-OFF-NEXT: [[TMP2:%.*]] = trunc <4 x i32> [[TMP1]] to <4 x i8>
+; STLF-OFF-NEXT: store <4 x i8> [[TMP2]], ptr [[S0]], align 1
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 64, %entry ], [ %i.next, %for.body ]
+ %bm16 = add i64 %i, -16
+ %bm12 = add i64 %i, -12
+ %bm8 = add i64 %i, -8
+ %bm4 = add i64 %i, -4
+ %p0 = getelementptr inbounds i8, ptr %A, i64 %bm16
+ %p1 = getelementptr inbounds i8, ptr %A, i64 %bm12
+ %p2 = getelementptr inbounds i8, ptr %A, i64 %bm8
+ %p3 = getelementptr inbounds i8, ptr %A, i64 %bm4
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+ %a0 = add i32 %l0, 1
+ %a1 = add i32 %l1, 1
+ %a2 = add i32 %l2, 1
+ %a3 = add i32 %l3, 1
+ %t0 = trunc i32 %a0 to i8
+ %t1 = trunc i32 %a1 to i8
+ %t2 = trunc i32 %a2 to i8
+ %t3 = trunc i32 %a3 to i8
+ %i1 = add i64 %i, 1
+ %i2 = add i64 %i, 2
+ %i3 = add i64 %i, 3
+ %s0 = getelementptr inbounds i8, ptr %A, i64 %i
+ %s1 = getelementptr inbounds i8, ptr %A, i64 %i1
+ %s2 = getelementptr inbounds i8, ptr %A, i64 %i2
+ %s3 = getelementptr inbounds i8, ptr %A, i64 %i3
+ store i8 %t0, ptr %s0, align 1
+ store i8 %t1, ptr %s1, align 1
+ store i8 %t2, ptr %s2, align 1
+ store i8 %t3, ptr %s3, align 1
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
>From dfe97eb0e691ef4943fcb2ed95b8d4be3813c0d5 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Fri, 21 Aug 2026 16:22:56 +0530
Subject: [PATCH 09/21] [SLP] Address review: width-aware STLF predicate and
penalty layering
- LAA: make term (a) width-aware (LoadElementSize > Distance % VectorStoreSize);
LAA passes the store width as the load width, so LAA behavior is unchanged.
- Keep one canonical MCSchedModel::DefaultStoreLoadForwardingPenalty (10) that
reaches targets via BasicTTIImpl; the TTI base now returns a neutral 0 like
getBranchMispredictPenalty(), dropping the duplicate constant and the
Analysis -> MC include. Effective penalty unchanged (Zen4 = 13).
- Collect candidate loop loads via a lambda so the set is const.
- Rework X86 STLF tests to widened-load recurrences and add an AArch64 test.
Co-authored-by: Cursor <cursoragent at cursor.com>
---
.../llvm/Analysis/LoopAccessAnalysis.h | 27 +-
.../llvm/Analysis/TargetTransformInfoImpl.h | 16 +-
llvm/include/llvm/MC/MCSchedule.h | 2 +-
llvm/lib/Analysis/LoopAccessAnalysis.cpp | 2 +-
.../Transforms/Vectorize/SLPVectorizer.cpp | 17 +-
.../AArch64/store-load-forward-conflict.ll | 187 +++++
.../X86/memory-runtime-checks.ll | 14 +-
.../X86/store-load-forward-conflict.ll | 715 +++++++-----------
.../X86/store-load-forward-cost.ll | 83 +-
.../X86/store-load-forward-non-power-of-2.ll | 117 +--
.../X86/store-load-forward-profitable.ll | 102 +--
11 files changed, 659 insertions(+), 623 deletions(-)
create mode 100644 llvm/test/Transforms/SLPVectorizer/AArch64/store-load-forward-conflict.ll
diff --git a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
index d021351f12164a..53d81e711bae02 100644
--- a/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
+++ b/llvm/include/llvm/Analysis/LoopAccessAnalysis.h
@@ -238,19 +238,21 @@ class MemoryDepChecker {
/// Returns true if a memory dependence at byte distance \p Distance between
/// a store and load would prevent store-to-load forwarding when the store is
/// widened to \p VectorStoreSize bytes. \p TypeByteSize is the store element
- /// size; \p LoadElementSize, when non-zero, is the load's access width.
+ /// size and \p LoadElementSize is the load's access width.
///
/// The conflicting store must still be likely to be in the store buffer
/// (\c Distance / VectorStoreSize is below 8 * TypeByteSize iterations), and
- /// the load must overlap two widened stores, which happens when either:
- /// (a) the load is misaligned w.r.t. the widened store window
- /// (\c Distance is not a multiple of \p VectorStoreSize), or
- /// (b) the load starts aligned but is itself wider than the widened store
- /// window (\p LoadElementSize > \p VectorStoreSize), so it overruns
- /// into the next widened store.
+ /// the load must overlap two widened stores. A backward load begins
+ /// \c R = \c Distance % \p VectorStoreSize bytes below a widened-store
+ /// boundary and overruns into the next widened store when either:
+ /// (a) it starts misaligned (\c R != 0) and is wider than the \c R bytes
+ /// left before that boundary (\p LoadElementSize > \c R), or
+ /// (b) it starts aligned (\c R == 0) but is itself wider than the widened
+ /// store window (\p LoadElementSize > \p VectorStoreSize).
+ /// A \p LoadElementSize of 0 (unknown width) disables both terms.
/// Both couldPreventStoreLoadForward and SLPVectorizer use this as their core
- /// STLF cost-model gate. LAA omits \p LoadElementSize (the load and store
- /// share the element size there, so (b) can never add anything).
+ /// STLF cost-model gate; LAA passes the widened store width as the load width,
+ /// so (a) reduces to "any misalignment conflicts" and (b) never fires.
static bool isStoreLoadForwardingConflict(uint64_t Distance,
uint64_t VectorStoreSize,
uint64_t TypeByteSize,
@@ -260,9 +262,10 @@ class MemoryDepChecker {
const uint64_t NumItersForStoreLoadThroughMemory = 8 * TypeByteSize;
if (Distance / VectorStoreSize >= NumItersForStoreLoadThroughMemory)
return false;
- // (a) Misaligned load: it straddles two widened stores.
- if (Distance % VectorStoreSize != 0)
- return true;
+ // (a) Misaligned load: it begins R bytes below a widened-store boundary and
+ // straddles two widened stores only if it is wider than those R bytes.
+ if (uint64_t R = Distance % VectorStoreSize)
+ return LoadElementSize > R;
// (b) Aligned load that is wider than the widened store still overruns
// into the next widened store (LoadElementSize == 0 disables this term).
return LoadElementSize > VectorStoreSize;
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 427fd5a93a6c4d..cfa4478d0f4b8f 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -681,19 +681,13 @@ class LLVM_ABI TargetTransformInfoImplBase {
virtual unsigned getCacheLineSize() const { return 0; }
- /// Default store-to-load forwarding stall penalty (in cycles) charged when no
- /// subtarget scheduling model is available. Kept intentionally in sync with
- /// MCSchedModel::DefaultStoreLoadForwardingPenalty (the scheduling-model
- /// field default); the cost layer keeps its own copy so it does not depend on
- /// MC.
- static constexpr unsigned DefaultStoreLoadForwardingPenalty = 10;
-
+ /// The store-to-load forwarding penalty is a microarchitectural cycle count.
+ /// Like getBranchMispredictPenalty, the target-independent base returns a
+ /// neutral 0; targets with a scheduling model supply the real value through
+ /// BasicTTIImpl (defaulting to MCSchedModel::DefaultStoreLoadForwardingPenalty).
virtual InstructionCost getStoreLoadForwardingConflictCost(
Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
- // No subtarget scheduling model is available here, so fall back to the
- // default penalty. Targets with a real scheduling model go through
- // BasicTTIImpl and use their own StoreLoadForwardingPenalty instead.
- return InstructionCost(DefaultStoreLoadForwardingPenalty);
+ return 0;
}
virtual std::optional<unsigned>
getCacheSize(TargetTransformInfo::CacheLevel Level) const {
diff --git a/llvm/include/llvm/MC/MCSchedule.h b/llvm/include/llvm/MC/MCSchedule.h
index a065123594bb0a..e3e32aed0edbdd 100644
--- a/llvm/include/llvm/MC/MCSchedule.h
+++ b/llvm/include/llvm/MC/MCSchedule.h
@@ -335,7 +335,7 @@ struct MCSchedModel {
// the default uses a conservative 10 cycles for targets that do not override
// it; individual targets may set a more precise value (e.g. Zen4 uses 13).
unsigned StoreLoadForwardingPenalty;
- static const unsigned DefaultStoreLoadForwardingPenalty = 10;
+ static constexpr unsigned DefaultStoreLoadForwardingPenalty = 10;
bool PostRAScheduler; // default value is false
diff --git a/llvm/lib/Analysis/LoopAccessAnalysis.cpp b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
index f5e57ea7c4d2a1..f05b1bd24a4f87 100644
--- a/llvm/lib/Analysis/LoopAccessAnalysis.cpp
+++ b/llvm/lib/Analysis/LoopAccessAnalysis.cpp
@@ -1934,7 +1934,7 @@ bool MemoryDepChecker::couldPreventStoreLoadForward(uint64_t Distance,
// shared with SLP via MemoryDepChecker::isStoreLoadForwardingConflict.
for (uint64_t VF = 2 * TypeByteSize;
VF <= MaxVFWithoutSLForwardIssuesPowerOf2; VF *= 2) {
- if (isStoreLoadForwardingConflict(Distance, VF, TypeByteSize)) {
+ if (isStoreLoadForwardingConflict(Distance, VF, TypeByteSize, VF)) {
MaxVFWithoutSLForwardIssuesPowerOf2 = (VF >> 1);
break;
}
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 2f85064f36966e..33f23c5e99c436 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -28334,13 +28334,16 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
// of the chain being costed. Enumerate every simple load in the store's loop
// that shares the store base.
Value *StoreBase = getUnderlyingObject(FirstStore->getPointerOperand());
- SmallPtrSet<LoadInst *, 8> CandidateLoads;
- for (BasicBlock *BB : StoreL->blocks())
- for (Instruction &I : *BB)
- if (auto *LoadI = dyn_cast<LoadInst>(&I))
- if (LoadI->isSimple() &&
- getUnderlyingObject(LoadI->getPointerOperand()) == StoreBase)
- CandidateLoads.insert(LoadI);
+ const auto CandidateLoads = [&] {
+ SmallPtrSet<LoadInst *, 8> Loads;
+ for (BasicBlock *BB : StoreL->blocks())
+ for (Instruction &I : *BB)
+ if (auto *LoadI = dyn_cast<LoadInst>(&I))
+ if (LoadI->isSimple() &&
+ getUnderlyingObject(LoadI->getPointerOperand()) == StoreBase)
+ Loads.insert(LoadI);
+ return Loads;
+ }();
if (CandidateLoads.empty())
return false;
diff --git a/llvm/test/Transforms/SLPVectorizer/AArch64/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/AArch64/store-load-forward-conflict.ll
new file mode 100644
index 00000000000000..11abe08cb7fe0b
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/AArch64/store-load-forward-conflict.ll
@@ -0,0 +1,187 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=slp-vectorizer -store-load-forwarding-conflict-cost=100 \
+; RUN: -S -mtriple=aarch64-- -mcpu=neoverse-v2 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-ON
+; RUN: opt < %s -passes=slp-vectorizer -store-load-forwarding-conflict-cost=0 \
+; RUN: -S -mtriple=aarch64-- -mcpu=neoverse-v2 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-OFF
+
+target datalayout = "e-m:e-i8:8:32-i16:16:32-i64:64-i128:128-n32:64-S128"
+target triple = "aarch64-unknown-linux-gnu"
+
+; The store-to-load forwarding cost model exercised on a non-X86 target. AArch64
+; subtargets do not set StoreLoadForwardingPenalty in their scheduling models, so
+; it resolves to the target-independent default
+; (MCSchedModel::DefaultStoreLoadForwardingPenalty = 10) that flows through
+; BasicTTIImpl -- the predicate does fire on AArch64. AArch64's vectorization
+; benefit for these small trees is larger than the default penalty, so we pin the
+; penalty with -store-load-forwarding-conflict-cost to make the cost effect
+; deterministic and observable, exercising the STLF cost path end-to-end for
+; AArch64 rather than only X86.
+
+; Widened backward load at a misaligned distance: the four consecutive loads
+; A[i-5..i-2] widen to a 16-byte <4 x i32> load, 20 bytes behind the 16-byte
+; <4 x i32> store at A[i] (20 % 16 = 4 -> misaligned; the 16-byte load overruns
+; the 4 bytes left before the boundary and straddles two widened stores). With a
+; non-zero penalty the store chain stays scalar; with a zero penalty it widens.
+define void @stlf_conflict_widened_misaligned(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @stlf_conflict_widened_misaligned(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; STLF-ON-NEXT: [[B1:%.*]] = add i64 [[I]], -4
+; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], -3
+; STLF-ON-NEXT: [[B3:%.*]] = add i64 [[I]], -2
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B1]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B3]]
+; STLF-ON-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[L0]], 1
+; STLF-ON-NEXT: [[T2:%.*]] = add nsw i32 [[L1]], 2
+; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[L2]], 3
+; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[L3]], 4
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: store i32 [[T2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP2]], align 4
+; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_conflict_widened_misaligned(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; STLF-OFF-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-OFF-NEXT: store <4 x i32> [[TMP1]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
+
+ %b0 = add i64 %i, -5
+ %b1 = add i64 %i, -4
+ %b2 = add i64 %i, -3
+ %b3 = add i64 %i, -2
+ %p0 = getelementptr inbounds i32, ptr %A, i64 %b0
+ %p1 = getelementptr inbounds i32, ptr %A, i64 %b1
+ %p2 = getelementptr inbounds i32, ptr %A, i64 %b2
+ %p3 = getelementptr inbounds i32, ptr %A, i64 %b3
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+
+ %t1 = add nsw i32 %l0, 1
+ %t2 = add nsw i32 %l1, 2
+ %t3 = add nsw i32 %l2, 3
+ %t4 = add nsw i32 %l3, 4
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+ %gep0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
+ %gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
+ %gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
+ store i32 %t1, ptr %gep0, align 4
+ store i32 %t2, ptr %gep1, align 4
+ store i32 %t3, ptr %gep2, align 4
+ store i32 %t4, ptr %gep3, align 4
+
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
+
+; Precision control: a single narrow scalar i32 load A[i-1], 4 bytes behind
+; (4 % 16 = 4 -> misaligned) but only 4 bytes wide, so it fits inside the bytes
+; left before the window boundary and never straddles two widened stores. No
+; forwarding conflict exists, so the chain widens even with the penalty pinned.
+define void @stlf_no_conflict_narrow_scalar_load(ptr noalias %A, i64 %n) {
+; CHECK-LABEL: define void @stlf_no_conflict_narrow_scalar_load(
+; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: br label %[[FOR_BODY:.*]]
+; CHECK: [[FOR_BODY]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
+; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
+; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
+; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
+; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
+; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; CHECK: [[FOR_END]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 1, %entry ], [ %i.next, %for.body ]
+
+ %back.idx = sub i64 %i, 1
+ %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
+ %t = load i32, ptr %back.gep, align 4
+
+ %t1 = add nsw i32 %t, 1
+ %t2 = add nsw i32 %t, 2
+ %t3 = add nsw i32 %t, 3
+ %t4 = add nsw i32 %t, 4
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+ %gep0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
+ %gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
+ %gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
+ store i32 %t1, ptr %gep0, align 4
+ store i32 %t2, ptr %gep1, align 4
+ store i32 %t3, ptr %gep2, align 4
+ store i32 %t4, ptr %gep3, align 4
+
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
index ccd69559b093f9..860405eb08a710 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
@@ -210,14 +210,12 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
; CHECK-NEXT: [[TMP24:%.*]] = getelementptr float, ptr [[ARG4:%.*]], i64 7
; CHECK-NEXT: br i1 [[C_2:%.*]], label [[BB25:%.*]], label [[BB22]]
; CHECK: bb25:
-; CHECK-NEXT: [[TMP26:%.*]] = getelementptr float, ptr [[ARG4]], i64 6
; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP24]], align 4
; CHECK-NEXT: [[TMP27:%.*]] = load float, ptr [[ARG5:%.*]], align 4
; CHECK-NEXT: [[TMP28:%.*]] = getelementptr float, ptr [[ARG4]], i64 5
; CHECK-NEXT: [[TMP29:%.*]] = fadd float 0.000000e+00, 0.000000e+00
-; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP26]], align 4
; CHECK-NEXT: [[TMP30:%.*]] = getelementptr float, ptr [[ARG4]], i64 4
-; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP28]], align 4
+; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP28]], align 4
; CHECK-NEXT: [[TMP31:%.*]] = fadd float 0.000000e+00, 0.000000e+00
; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP30]], align 4
; CHECK-NEXT: br label [[BB33:%.*]]
@@ -225,17 +223,13 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
; CHECK-NEXT: br label [[BB34:%.*]]
; CHECK: bb34:
; CHECK-NEXT: [[TMP35:%.*]] = getelementptr float, ptr [[ARG4]], i64 3
-; CHECK-NEXT: [[ARG3:%.*]] = getelementptr float, ptr [[ARG4]], i64 2
; CHECK-NEXT: [[TMP37:%.*]] = load float, ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP38:%.*]] = fadd float 0.000000e+00, [[TMP37]]
; CHECK-NEXT: store float [[TMP38]], ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP39:%.*]] = getelementptr float, ptr [[ARG4]], i64 1
-; CHECK-NEXT: [[TMP44:%.*]] = load float, ptr [[ARG3]], align 4
-; CHECK-NEXT: [[TMP41:%.*]] = fadd float 0.000000e+00, [[TMP44]]
-; CHECK-NEXT: store float [[TMP41]], ptr [[ARG3]], align 4
-; CHECK-NEXT: [[TMP45:%.*]] = load float, ptr [[TMP39]], align 4
-; CHECK-NEXT: [[TMP46:%.*]] = fadd float 0.000000e+00, [[TMP45]]
-; CHECK-NEXT: store float [[TMP46]], ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP5:%.*]] = load <2 x float>, ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP7:%.*]] = fadd <2 x float> zeroinitializer, [[TMP5]]
+; CHECK-NEXT: store <2 x float> [[TMP7]], ptr [[TMP39]], align 4
; CHECK-NEXT: [[TMP47:%.*]] = load float, ptr [[ARG6:%.*]], align 4
; CHECK-NEXT: [[TMP48:%.*]] = load float, ptr [[ARG4]], align 4
; CHECK-NEXT: [[TMP49:%.*]] = fadd float 0.000000e+00, [[TMP48]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index 6ac6afbff18cf6..55d1c956ad21f4 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -9,23 +9,16 @@ target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:
target triple = "x86_64-unknown-linux-gnu"
;
-; Test 1: Backward loop-carried load at misaligned distance.
+; Test 1: Widened backward load at a misaligned distance.
;
-; for (int i = 5; i < n; i += 4) {
-; int t = A[i-5]; ; load 20 bytes (5*4) before store base
-; A[i] = t + 1;
-; A[i+1] = t + 2;
-; A[i+2] = t + 3;
-; A[i+3] = t + 4;
-; }
-;
-; After SLP with VF=4, the four stores would become a single 16-byte vector
-; store at A[i]. The load at A[i-5] is 20 bytes behind:
-; 20 % 16 = 4 -> misaligned, load straddles two pending vector stores
-; 20 / 16 = 1 -> store still hot in store buffer
-; The STLF penalty makes the VF=4 store entry unprofitable. Sub-chains at
-; VF=2 are priced per-base: {A[i+1],A[i+2]} has distance 24, 24%8=0 -> no
-; conflict, no penalty, so it still vectorizes.
+; The four consecutive loads A[i-5..i-2] widen to a 16-byte <4 x i32> load; the
+; four stores A[i..i+3] widen to a 16-byte <4 x i32> store. The load base is 20
+; bytes behind:
+; 20 % 16 = 4 -> misaligned; the 16-byte load overruns the 4 bytes left before
+; the window boundary, straddling two widened stores.
+; 20 / 16 = 1 -> store still hot in the store buffer.
+; So the STLF penalty makes the VF=4 store entry unprofitable and it stays scalar
+; with the check on; with the check off it widens.
;
define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
; STLF-ON-LABEL: define void @stlf_conflict_backward_misaligned(
@@ -33,23 +26,26 @@ define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[ENTRY:.*]]:
; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
; STLF-ON: [[FOR_BODY]]:
-; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 5, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
-; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; STLF-ON-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
-; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
-; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
-; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[TMP2:%.*]] = extractelement <4 x i32> [[TMP1]], i64 0
+; STLF-ON-NEXT: store i32 [[TMP2]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: [[TMP3:%.*]] = extractelement <4 x i32> [[TMP1]], i64 1
+; STLF-ON-NEXT: store i32 [[TMP3]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: [[TMP4:%.*]] = extractelement <4 x i32> [[TMP1]], i64 2
+; STLF-ON-NEXT: store i32 [[TMP4]], ptr [[GEP2]], align 4
+; STLF-ON-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[TMP1]], i64 3
+; STLF-ON-NEXT: store i32 [[TMP5]], ptr [[GEP3]], align 4
; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -61,15 +57,13 @@ define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
; STLF-OFF-NEXT: [[ENTRY:.*]]:
; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
; STLF-OFF: [[FOR_BODY]]:
-; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 5, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-OFF-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; STLF-OFF-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-OFF-NEXT: store <4 x i32> [[TMP1]], ptr [[GEP0]], align 4
; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -80,26 +74,33 @@ entry:
br label %for.body
for.body:
- %i = phi i64 [ 5, %entry ], [ %i.next, %for.body ]
-
- %back.idx = sub i64 %i, 5
- %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
- %t = load i32, ptr %back.gep, align 4
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
+ %b0 = add i64 %i, -5
+ %b1 = add i64 %i, -4
+ %b2 = add i64 %i, -3
+ %b3 = add i64 %i, -2
+ %p0 = getelementptr inbounds i32, ptr %A, i64 %b0
+ %p1 = getelementptr inbounds i32, ptr %A, i64 %b1
+ %p2 = getelementptr inbounds i32, ptr %A, i64 %b2
+ %p3 = getelementptr inbounds i32, ptr %A, i64 %b3
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+
+ %t1 = add nsw i32 %l0, 1
+ %t2 = add nsw i32 %l1, 2
+ %t3 = add nsw i32 %l2, 3
+ %t4 = add nsw i32 %l3, 4
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
@@ -114,25 +115,22 @@ for.end:
}
;
-; Test 2: Backward load at distance ALIGNED to vector store width.
-;
-; for (int i = 4; i < n; i += 4) {
-; int t = A[i-4]; ; load 16 bytes (4*4) before store base
-; A[i] = t + 1; ...
-; }
+; Test 2: Narrow scalar load at a misaligned distance (precision control).
;
-; Distance = 16, VectorStoreBytes = 16, 16 % 16 == 0 -> NOT misaligned.
-; The load lands exactly on a previous vector store, so STLF can forward.
-; The check should NOT fire; vectorization is allowed.
+; A single scalar i32 load A[i-1] broadcast to the four stores. It is 4 bytes
+; behind (4 % 16 = 4 -> misaligned) but the load is only 4 bytes wide, so it
+; fits entirely inside the 4 bytes left before the window boundary and never
+; straddles two widened stores. A 4-byte load can never overrun a 16-byte
+; window, so forwarding succeeds: the check must NOT fire and the chain widens.
;
-define void @stlf_no_conflict_backward_aligned(ptr noalias %A, i64 %n) {
-; CHECK-LABEL: define void @stlf_no_conflict_backward_aligned(
+define void @stlf_no_conflict_narrow_scalar_load(ptr noalias %A, i64 %n) {
+; CHECK-LABEL: define void @stlf_no_conflict_narrow_scalar_load(
; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: br label %[[FOR_BODY:.*]]
; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 4, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 4
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
@@ -150,9 +148,9 @@ entry:
br label %for.body
for.body:
- %i = phi i64 [ 4, %entry ], [ %i.next, %for.body ]
+ %i = phi i64 [ 1, %entry ], [ %i.next, %for.body ]
- %back.idx = sub i64 %i, 4
+ %back.idx = sub i64 %i, 1
%back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
%t = load i32, ptr %back.gep, align 4
@@ -164,12 +162,10 @@ for.body:
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
@@ -186,8 +182,9 @@ for.end:
;
; Test 3: Forward reference (load AHEAD of stores).
;
-; Positive pointer diff -> not a backward loop-carried dep, no STLF concern.
-; Vectorization should proceed.
+; A widened load A[i+64..i+67] has a positive pointer diff, so it is not a
+; backward loop-carried dependence and cannot cause STLF on the widened store.
+; Vectorization should proceed under both configurations.
;
define void @stlf_no_conflict_forward(ptr noalias %A, i64 %n) {
; CHECK-LABEL: define void @stlf_no_conflict_forward(
@@ -196,14 +193,12 @@ define void @stlf_no_conflict_forward(ptr noalias %A, i64 %n) {
; CHECK-NEXT: br label %[[FOR_BODY:.*]]
; CHECK: [[FOR_BODY]]:
; CHECK-NEXT: [[I:%.*]] = phi i64 [ 0, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[FWD_IDX:%.*]] = add nuw nsw i64 [[I]], 64
-; CHECK-NEXT: [[FWD_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[FWD_IDX]]
-; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[FWD_GEP]], align 4
+; CHECK-NEXT: [[F0:%.*]] = add nuw nsw i64 [[I]], 64
+; CHECK-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[F0]]
; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; CHECK-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; CHECK-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; CHECK-NEXT: store <4 x i32> [[TMP1]], ptr [[GEP0]], align 4
; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -216,24 +211,31 @@ entry:
for.body:
%i = phi i64 [ 0, %entry ], [ %i.next, %for.body ]
- %fwd.idx = add nuw nsw i64 %i, 64
- %fwd.gep = getelementptr inbounds i32, ptr %A, i64 %fwd.idx
- %t = load i32, ptr %fwd.gep, align 4
-
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
+ %f0 = add nuw nsw i64 %i, 64
+ %f1 = add nuw nsw i64 %i, 65
+ %f2 = add nuw nsw i64 %i, 66
+ %f3 = add nuw nsw i64 %i, 67
+ %p0 = getelementptr inbounds i32, ptr %A, i64 %f0
+ %p1 = getelementptr inbounds i32, ptr %A, i64 %f1
+ %p2 = getelementptr inbounds i32, ptr %A, i64 %f2
+ %p3 = getelementptr inbounds i32, ptr %A, i64 %f3
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+
+ %t1 = add nsw i32 %l0, 1
+ %t2 = add nsw i32 %l1, 2
+ %t3 = add nsw i32 %l2, 3
+ %t4 = add nsw i32 %l3, 4
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
@@ -250,8 +252,8 @@ for.end:
;
; Test 4: Stores not in a loop (straight-line code).
;
-; STLF stalls only matter when the same code executes repeatedly and the
-; store buffer is hot. Outside loops the check should not fire.
+; STLF stalls only matter when the same code executes repeatedly and the store
+; buffer is hot. Outside loops the check should not fire.
;
define void @stlf_no_conflict_no_loop(ptr noalias %A) {
; CHECK-LABEL: define void @stlf_no_conflict_no_loop(
@@ -286,56 +288,60 @@ define void @stlf_no_conflict_no_loop(ptr noalias %A) {
;
; Test 5: Load from a DIFFERENT underlying object.
;
-; The check restricts scanning to loads with the same underlying object as
-; the stores. A load from an unrelated array can never cause STLF on the
-; widened store, so the check should not fire.
+; The check restricts scanning to loads with the same underlying object as the
+; stores. A widened load from an unrelated array B can never cause STLF on the
+; widened store to A, so the check should not fire.
;
-define void @stlf_no_conflict_different_base(ptr noalias %A, ptr noalias %B,
+define void @stlf_no_conflict_different_base(ptr noalias %A, ptr noalias %B, i64 %n) {
; CHECK-LABEL: define void @stlf_no_conflict_different_base(
; CHECK-SAME: ptr noalias [[A:%.*]], ptr noalias [[B:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
; CHECK-NEXT: [[ENTRY:.*]]:
; CHECK-NEXT: br label %[[FOR_BODY:.*]]
; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 5, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[B_IDX:%.*]] = sub i64 [[I]], 5
-; CHECK-NEXT: [[B_GEP:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[B_IDX]]
-; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[B_GEP]], align 4
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; CHECK-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[B]], i64 [[B0]]
; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; CHECK-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; CHECK-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; CHECK-NEXT: store <4 x i32> [[TMP1]], ptr [[GEP0]], align 4
; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
; CHECK: [[FOR_END]]:
; CHECK-NEXT: ret void
;
- i64 %n) {
entry:
br label %for.body
for.body:
- %i = phi i64 [ 5, %entry ], [ %i.next, %for.body ]
-
- %b.idx = sub i64 %i, 5
- %b.gep = getelementptr inbounds i32, ptr %B, i64 %b.idx
- %t = load i32, ptr %b.gep, align 4
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
+ %b0 = add i64 %i, -5
+ %b1 = add i64 %i, -4
+ %b2 = add i64 %i, -3
+ %b3 = add i64 %i, -2
+ %p0 = getelementptr inbounds i32, ptr %B, i64 %b0
+ %p1 = getelementptr inbounds i32, ptr %B, i64 %b1
+ %p2 = getelementptr inbounds i32, ptr %B, i64 %b2
+ %p3 = getelementptr inbounds i32, ptr %B, i64 %b3
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+
+ %t1 = add nsw i32 %l0, 1
+ %t2 = add nsw i32 %l1, 2
+ %t3 = add nsw i32 %l2, 3
+ %t4 = add nsw i32 %l3, 4
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
@@ -352,9 +358,9 @@ for.end:
;
; Test 6: Volatile load is skipped (not "simple").
;
-; A volatile load fails LoadInst::isSimple() and is never considered by the
-; STLF check, even at a clearly conflicting distance. The check therefore
-; finds no offending load and vectorization proceeds.
+; A volatile load fails LoadInst::isSimple() and is never considered by the STLF
+; check, even at a clearly conflicting distance. The check finds no offending
+; load and vectorization proceeds.
;
define void @stlf_volatile_load_skipped(ptr noalias %A, i64 %n) {
; CHECK-LABEL: define void @stlf_volatile_load_skipped(
@@ -395,12 +401,10 @@ for.body:
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
@@ -415,102 +419,65 @@ for.end:
}
;
-; Test 7: Loads that feed scalar arithmetic below a gather/splat leaf.
+; Test 7: Distance past the recency (NumItersForSafety) boundary.
;
-; A[i-5] and A[i-7] are misaligned to a 16-byte vector store (distances 20 and
-; 28 bytes, both %16 != 0) but only feed the scalar %sum, which is broadcast, so
-; they never become load tree nodes of the chain. Candidate loads are now
-; enumerated from all simple loads in the store's loop, not just tree nodes, so
-; these loads are seen and the chain is kept scalar under STLF-ON.
+; For i32 with VF=4: TypeByteSize=4, VectorStoreBytes=16, the store is assumed
+; hot only while Distance / 16 < 8 * 4 = 32. A widened load whose nearest element
+; is 128 elements back has all four element distances 524/520/516/512 bytes, each
+; 512/16 = 32 iterations away (NOT < 32), so the store is no longer in the buffer
+; even though the load is wide and (some elements) misaligned. The check does not
+; fire and vectorization proceeds.
;
-define void @stlf_multiple_conflicting_loads(ptr noalias %A, i64 %n) {
-; STLF-ON-LABEL: define void @stlf_multiple_conflicting_loads(
-; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-ON-NEXT: [[ENTRY:.*]]:
-; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-ON: [[FOR_BODY]]:
-; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-ON-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-ON-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
-; STLF-ON-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
-; STLF-ON-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
-; STLF-ON-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
-; STLF-ON-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
-; STLF-ON-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
-; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[SUM]], 1
-; STLF-ON-NEXT: [[T2:%.*]] = add nsw i32 [[SUM]], 2
-; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[SUM]], 3
-; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[SUM]], 4
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
-; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
-; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
-; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
-; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
-; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: store i32 [[T2]], ptr [[GEP1]], align 4
-; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP2]], align 4
-; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
-; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-ON: [[FOR_END]]:
-; STLF-ON-NEXT: ret void
-;
-; STLF-OFF-LABEL: define void @stlf_multiple_conflicting_loads(
-; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-OFF-NEXT: [[ENTRY:.*]]:
-; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-OFF: [[FOR_BODY]]:
-; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-OFF-NEXT: [[BACK5_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-OFF-NEXT: [[BACK5_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK5_IDX]]
-; STLF-OFF-NEXT: [[T5:%.*]] = load i32, ptr [[BACK5_GEP]], align 4
-; STLF-OFF-NEXT: [[BACK7_IDX:%.*]] = sub i64 [[I]], 7
-; STLF-OFF-NEXT: [[BACK7_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK7_IDX]]
-; STLF-OFF-NEXT: [[T7:%.*]] = load i32, ptr [[BACK7_GEP]], align 4
-; STLF-OFF-NEXT: [[SUM:%.*]] = add nsw i32 [[T5]], [[T7]]
-; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[SUM]], i64 0
-; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-OFF: [[FOR_END]]:
-; STLF-OFF-NEXT: ret void
+define void @stlf_no_conflict_iter_safety_boundary(ptr noalias %A, i64 %n) {
+; CHECK-LABEL: define void @stlf_no_conflict_iter_safety_boundary(
+; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: br label %[[FOR_BODY:.*]]
+; CHECK: [[FOR_BODY]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 131, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; CHECK-NEXT: [[B0:%.*]] = add i64 [[I]], -131
+; CHECK-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; CHECK-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; CHECK-NEXT: store <4 x i32> [[TMP1]], ptr [[GEP0]], align 4
+; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; CHECK: [[FOR_END]]:
+; CHECK-NEXT: ret void
;
entry:
br label %for.body
for.body:
- %i = phi i64 [ 7, %entry ], [ %i.next, %for.body ]
-
- %back5.idx = sub i64 %i, 5
- %back5.gep = getelementptr inbounds i32, ptr %A, i64 %back5.idx
- %t5 = load i32, ptr %back5.gep, align 4
-
- %back7.idx = sub i64 %i, 7
- %back7.gep = getelementptr inbounds i32, ptr %A, i64 %back7.idx
- %t7 = load i32, ptr %back7.gep, align 4
-
- %sum = add nsw i32 %t5, %t7
- %t1 = add nsw i32 %sum, 1
- %t2 = add nsw i32 %sum, 2
- %t3 = add nsw i32 %sum, 3
- %t4 = add nsw i32 %sum, 4
+ %i = phi i64 [ 131, %entry ], [ %i.next, %for.body ]
+
+ %b0 = add i64 %i, -131
+ %b1 = add i64 %i, -130
+ %b2 = add i64 %i, -129
+ %b3 = add i64 %i, -128
+ %p0 = getelementptr inbounds i32, ptr %A, i64 %b0
+ %p1 = getelementptr inbounds i32, ptr %A, i64 %b1
+ %p2 = getelementptr inbounds i32, ptr %A, i64 %b2
+ %p3 = getelementptr inbounds i32, ptr %A, i64 %b3
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+
+ %t1 = add nsw i32 %l0, 1
+ %t2 = add nsw i32 %l1, 2
+ %t3 = add nsw i32 %l2, 3
+ %t4 = add nsw i32 %l3, 4
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
@@ -528,9 +495,9 @@ for.end:
; Test 8: getPointersDiff cannot determine a constant offset.
;
; The conflicting load uses a runtime index loaded from a separate array, so
-; SCEV cannot determine the byte offset relative to the stores.
-; getPointersDiff returns nullopt, the check skips this load, finds no
-; conflict, and vectorization proceeds.
+; SCEV cannot determine the byte offset relative to the stores. getPointersDiff
+; returns nullopt, the check skips this load, finds no conflict, and
+; vectorization proceeds.
;
define void @stlf_pointer_diff_unknown(ptr noalias %A, ptr noalias %IdxA, i64 %n) {
; CHECK-LABEL: define void @stlf_pointer_diff_unknown(
@@ -573,175 +540,10 @@ for.body:
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
- %gep0 = getelementptr inbounds i32, ptr %A, i64 %i
- %gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
- %gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
- %gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
- store i32 %t1, ptr %gep0, align 4
- store i32 %t2, ptr %gep1, align 4
- store i32 %t3, ptr %gep2, align 4
- store i32 %t4, ptr %gep3, align 4
-
- %i.next = add nuw nsw i64 %i, 4
- %cmp = icmp slt i64 %i.next, %n
- br i1 %cmp, label %for.body, label %for.end
-
-for.end:
- ret void
-}
-
-;
-; Test 9: Motivating example -- a[i] = a[i-1] + 1.
-;
-; Distance = 4 bytes (1 element) from chain base. VectorStoreBytes = 16
-; at VF=4. 4 % 16 = 4 -> misaligned -> STLF conflict. The full 4-wide store
-; entry is penalized and is not profitable. Sub-chains that are individually
-; misaligned are penalized too; only sub-chains where the distance happens to
-; be aligned to the narrower vector store width vectorize.
-;
-define void @stlf_conflict_short_backward(ptr noalias %A, i64 %n) {
-; STLF-ON-LABEL: define void @stlf_conflict_short_backward(
-; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-ON-NEXT: [[ENTRY:.*]]:
-; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-ON: [[FOR_BODY]]:
-; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
-; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
-; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
-; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
-; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
-; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
-; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
-; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
-; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
-; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-ON: [[FOR_END]]:
-; STLF-ON-NEXT: ret void
-;
-; STLF-OFF-LABEL: define void @stlf_conflict_short_backward(
-; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-OFF-NEXT: [[ENTRY:.*]]:
-; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-OFF: [[FOR_BODY]]:
-; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
-; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-OFF-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; STLF-OFF-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-OFF: [[FOR_END]]:
-; STLF-OFF-NEXT: ret void
-;
-entry:
- br label %for.body
-
-for.body:
- %i = phi i64 [ 1, %entry ], [ %i.next, %for.body ]
-
- %back.idx = sub i64 %i, 1
- %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
- %t = load i32, ptr %back.gep, align 4
-
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
-
- %i1 = add nuw nsw i64 %i, 1
- %i2 = add nuw nsw i64 %i, 2
- %i3 = add nuw nsw i64 %i, 3
-
- %gep0 = getelementptr inbounds i32, ptr %A, i64 %i
- %gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
- %gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
- %gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
- store i32 %t1, ptr %gep0, align 4
- store i32 %t2, ptr %gep1, align 4
- store i32 %t3, ptr %gep2, align 4
- store i32 %t4, ptr %gep3, align 4
-
- %i.next = add nuw nsw i64 %i, 4
- %cmp = icmp slt i64 %i.next, %n
- br i1 %cmp, label %for.body, label %for.end
-
-for.end:
- ret void
-}
-
-;
-; Test 10: Distance just past NumItersForSafety boundary.
-;
-; For i32 with VF=4: ElementSize=4, VectorStoreBytes=16,
-; NumItersForSafety = 8 * ElementSize = 32. The check fires only when
-; Distance / VectorStoreBytes < 32. With diff = -129 (load 129 elements
-; back), Distance = 516 bytes, 516 / 16 = 32 (NOT < 32), so the recency
-; predicate fails and the check does NOT fire even though 516 % 16 = 4
-; (misaligned). Vectorization proceeds.
-;
-define void @stlf_no_conflict_iter_safety_boundary(ptr noalias %A, i64 %n) {
-; CHECK-LABEL: define void @stlf_no_conflict_iter_safety_boundary(
-; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; CHECK-NEXT: [[ENTRY:.*]]:
-; CHECK-NEXT: br label %[[FOR_BODY:.*]]
-; CHECK: [[FOR_BODY]]:
-; CHECK-NEXT: [[I:%.*]] = phi i64 [ 129, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; CHECK-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 129
-; CHECK-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; CHECK-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; CHECK-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; CHECK-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; CHECK-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; CHECK-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; CHECK-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; CHECK-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
-; CHECK-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; CHECK-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; CHECK: [[FOR_END]]:
-; CHECK-NEXT: ret void
-;
-entry:
- br label %for.body
-
-for.body:
- %i = phi i64 [ 129, %entry ], [ %i.next, %for.body ]
-
- %back.idx = sub i64 %i, 129
- %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
- %t = load i32, ptr %back.gep, align 4
-
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
-
- %i1 = add nuw nsw i64 %i, 1
- %i2 = add nuw nsw i64 %i, 2
- %i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
@@ -756,93 +558,13 @@ for.end:
}
;
-; Test 11: Non-i32 element type (i64 VF=2).
-;
-; Verifies the byte-distance math works for wider types. With i64
-; (ElementSize=8) and VF=2: VectorStoreBytes=16, NumItersForSafety=64.
-; A load 3 elements back has Distance = 24 bytes, 24/16=1 < 64, and
-; 24%16=8 != 0, so the check fires.
+; Test 9: Aligned load that is WIDER than the widened store (crossing).
;
-define void @stlf_conflict_backward_misaligned_i64(ptr noalias %A, i64 %n) {
-; STLF-ON-LABEL: define void @stlf_conflict_backward_misaligned_i64(
-; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-ON-NEXT: [[ENTRY:.*]]:
-; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-ON: [[FOR_BODY]]:
-; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 3, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 3
-; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-ON-NEXT: [[T:%.*]] = load i64, ptr [[BACK_GEP]], align 8
-; STLF-ON-NEXT: [[T1:%.*]] = add nsw i64 [[T]], 1
-; STLF-ON-NEXT: [[T2:%.*]] = add nsw i64 [[T]], 2
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
-; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I]]
-; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I1]]
-; STLF-ON-NEXT: store i64 [[T1]], ptr [[GEP0]], align 8
-; STLF-ON-NEXT: store i64 [[T2]], ptr [[GEP1]], align 8
-; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 2
-; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-ON: [[FOR_END]]:
-; STLF-ON-NEXT: ret void
-;
-; STLF-OFF-LABEL: define void @stlf_conflict_backward_misaligned_i64(
-; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
-; STLF-OFF-NEXT: [[ENTRY:.*]]:
-; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
-; STLF-OFF: [[FOR_BODY]]:
-; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 3, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 3
-; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-OFF-NEXT: [[T:%.*]] = load i64, ptr [[BACK_GEP]], align 8
-; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I]]
-; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <2 x i64> poison, i64 [[T]], i64 0
-; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <2 x i64> [[TMP0]], <2 x i64> poison, <2 x i32> zeroinitializer
-; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <2 x i64> [[TMP1]], <i64 1, i64 2>
-; STLF-OFF-NEXT: store <2 x i64> [[TMP2]], ptr [[GEP0]], align 8
-; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 2
-; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
-; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
-; STLF-OFF: [[FOR_END]]:
-; STLF-OFF-NEXT: ret void
-;
-entry:
- br label %for.body
-
-for.body:
- %i = phi i64 [ 3, %entry ], [ %i.next, %for.body ]
-
- %back.idx = sub i64 %i, 3
- %back.gep = getelementptr inbounds i64, ptr %A, i64 %back.idx
- %t = load i64, ptr %back.gep, align 8
-
- %t1 = add nsw i64 %t, 1
- %t2 = add nsw i64 %t, 2
-
- %i1 = add nuw nsw i64 %i, 1
-
- %gep0 = getelementptr inbounds i64, ptr %A, i64 %i
- %gep1 = getelementptr inbounds i64, ptr %A, i64 %i1
-
- store i64 %t1, ptr %gep0, align 8
- store i64 %t2, ptr %gep1, align 8
-
- %i.next = add nuw nsw i64 %i, 2
- %cmp = icmp slt i64 %i.next, %n
- br i1 %cmp, label %for.body, label %for.end
-
-for.end:
- ret void
-}
-
-;
-; Test 12: Aligned load that is WIDER than the widened store (crossing).
-;
-; i8 store chain, VF=4 => widened store window W = 4 bytes. A backward i64
-; load (element size 8 > 4) sits at aligned Distance = 4: 4 % 4 == 0 (not
-; misaligned), but the load itself overruns its window and straddles two
-; widened stores. The misalignment-only test would miss this; the load-width
-; term catches it, so vectorization must be rejected with the check on.
+; i8 store chain, VF=4 => widened store window W = 4 bytes. A backward i64 load
+; (element size 8 > 4) sits at aligned Distance = 4: 4 % 4 == 0 (not misaligned),
+; but the load itself overruns its window and straddles two widened stores. The
+; misalignment-only term (a) would miss this; the load-width term (b) catches it,
+; so vectorization must be rejected with the check on.
;
define void @stlf_conflict_cross_boundary_wider_load(ptr noalias %A, i64 %n) {
; STLF-ON-LABEL: define void @stlf_conflict_cross_boundary_wider_load(
@@ -963,13 +685,13 @@ for.end:
}
;
-; Test 13: Aligned load wider than the element but CONTAINED (no crossing).
+; Test 10: Aligned load wider than the element but CONTAINED (no crossing).
;
-; Same shape as Test 12, but the backward load is i32 (element size 4 == W).
+; Same shape as Test 9, but the backward load is i32 (element size 4 == W).
; Starting aligned, it exactly fills one widened store window and does not
; overrun into the next one, so it is NOT a forwarding conflict. Vectorization
-; must happen regardless of the STLF check, proving the load-width term does
-; not over-reject aligned, contained loads.
+; must happen regardless of the STLF check, proving the load-width term does not
+; over-reject aligned, contained loads.
;
define void @stlf_no_conflict_wider_contained_load(ptr noalias %A, i64 %n) {
; CHECK-LABEL: define void @stlf_no_conflict_wider_contained_load(
@@ -1045,3 +767,88 @@ for.body:
for.end:
ret void
}
+
+;
+; Test 11: Widened backward load at a misaligned distance, i64 (VF=2).
+;
+; Verifies the byte-distance math for wider types. Two consecutive i64 loads
+; A[i-3..i-2] widen to a 16-byte <2 x i64> load; the stores A[i..i+1] widen to a
+; 16-byte <2 x i64> store. The load base is 24 bytes behind:
+; 24 % 16 = 8 -> misaligned; the 16-byte load overruns the 8 bytes left
+; before the window boundary and straddles two widened stores.
+; 24 / 16 = 1 -> store still hot.
+; So the check fires with the check on and the store stays scalar.
+;
+define void @stlf_conflict_backward_misaligned_i64(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @stlf_conflict_backward_misaligned_i64(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -3
+; STLF-ON-NEXT: [[B1:%.*]] = add i64 [[I]], -2
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[B0]]
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[B1]]
+; STLF-ON-NEXT: [[L0:%.*]] = load i64, ptr [[P0]], align 8
+; STLF-ON-NEXT: [[L1:%.*]] = load i64, ptr [[P1]], align 8
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i64 [[L0]], 1
+; STLF-ON-NEXT: [[T2:%.*]] = add nsw i64 [[L1]], 2
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: store i64 [[T1]], ptr [[GEP0]], align 8
+; STLF-ON-NEXT: store i64 [[T2]], ptr [[GEP1]], align 8
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @stlf_conflict_backward_misaligned_i64(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[B0:%.*]] = add i64 [[I]], -3
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[B0]]
+; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i64, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = load <2 x i64>, ptr [[P0]], align 8
+; STLF-OFF-NEXT: [[TMP1:%.*]] = add nsw <2 x i64> [[TMP0]], <i64 1, i64 2>
+; STLF-OFF-NEXT: store <2 x i64> [[TMP1]], ptr [[GEP0]], align 8
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 2
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
+
+ %b0 = add i64 %i, -3
+ %b1 = add i64 %i, -2
+ %p0 = getelementptr inbounds i64, ptr %A, i64 %b0
+ %p1 = getelementptr inbounds i64, ptr %A, i64 %b1
+ %l0 = load i64, ptr %p0, align 8
+ %l1 = load i64, ptr %p1, align 8
+
+ %t1 = add nsw i64 %l0, 1
+ %t2 = add nsw i64 %l1, 2
+
+ %i1 = add nuw nsw i64 %i, 1
+ %gep0 = getelementptr inbounds i64, ptr %A, i64 %i
+ %gep1 = getelementptr inbounds i64, ptr %A, i64 %i1
+ store i64 %t1, ptr %gep0, align 8
+ store i64 %t2, ptr %gep1, align 8
+
+ %i.next = add nuw nsw i64 %i, 2
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
index 56210cfca11b15..74921709bbaf4f 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
@@ -1,11 +1,11 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
; Store-to-load forwarding (STLF) is priced into the SLP cost model rather than
-; vetoing vectorization outright. The motivating loop a[i] = a[i-1] + 1 has a
-; misaligned backward dependence that breaks forwarding once the four scalar
-; stores become one 16-byte vector store.
+; vetoing vectorization outright. A widened backward load that is misaligned to
+; the widened store window breaks forwarding once the four scalar stores become
+; one 16-byte vector store.
;
; With the target's default STLF penalty the small tree is not worth the stall,
-; so it stays (mostly) scalar. Zeroing the penalty via the tuning knob lets the
+; so the stores stay scalar. Zeroing the penalty via the tuning knob lets the
; very same tree vectorize to <4 x i32> -- demonstrating this is a cost, not a
; hard bail-out, so a sufficiently profitable tree can still vectorize.
;
@@ -18,29 +18,37 @@
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"
+; The four consecutive backward loads A[i-5..i-2] widen to a 16-byte <4 x i32>
+; load; the stores A[i..i+3] widen to a 16-byte <4 x i32> store. The load base
+; is 20 bytes behind (20 % 16 = 4 -> misaligned; the 16-byte load overruns the
+; 4 bytes left before the window boundary, so it straddles two widened stores)
+; and 20 / 16 = 1 iteration back, still hot in the store buffer.
define void @stlf_recurrence(ptr noalias %A, i64 %n) {
; PENALTY-LABEL: define void @stlf_recurrence(
; PENALTY-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
; PENALTY-NEXT: [[ENTRY:.*]]:
; PENALTY-NEXT: br label %[[FOR_BODY:.*]]
; PENALTY: [[FOR_BODY]]:
-; PENALTY-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; PENALTY-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
-; PENALTY-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; PENALTY-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; PENALTY-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
-; PENALTY-NEXT: [[T4:%.*]] = add nsw i32 [[T]], 4
-; PENALTY-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
+; PENALTY-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; PENALTY-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; PENALTY-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; PENALTY-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; PENALTY-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; PENALTY-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; PENALTY-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
; PENALTY-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; PENALTY-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; PENALTY-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; PENALTY-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
; PENALTY-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; PENALTY-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
-; PENALTY-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; PENALTY-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
-; PENALTY-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; PENALTY-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
-; PENALTY-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; PENALTY-NEXT: [[TMP2:%.*]] = extractelement <4 x i32> [[TMP1]], i64 0
+; PENALTY-NEXT: store i32 [[TMP2]], ptr [[GEP0]], align 4
+; PENALTY-NEXT: [[TMP3:%.*]] = extractelement <4 x i32> [[TMP1]], i64 1
+; PENALTY-NEXT: store i32 [[TMP3]], ptr [[GEP1]], align 4
+; PENALTY-NEXT: [[TMP4:%.*]] = extractelement <4 x i32> [[TMP1]], i64 2
+; PENALTY-NEXT: store i32 [[TMP4]], ptr [[GEP2]], align 4
+; PENALTY-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[TMP1]], i64 3
+; PENALTY-NEXT: store i32 [[TMP5]], ptr [[GEP3]], align 4
; PENALTY-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; PENALTY-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; PENALTY-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -52,15 +60,13 @@ define void @stlf_recurrence(ptr noalias %A, i64 %n) {
; NOPENALTY-NEXT: [[ENTRY:.*]]:
; NOPENALTY-NEXT: br label %[[FOR_BODY:.*]]
; NOPENALTY: [[FOR_BODY]]:
-; NOPENALTY-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; NOPENALTY-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 1
-; NOPENALTY-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; NOPENALTY-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; NOPENALTY-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; NOPENALTY-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; NOPENALTY-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
; NOPENALTY-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; NOPENALTY-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; NOPENALTY-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; NOPENALTY-NEXT: [[TMP2:%.*]] = add nsw <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; NOPENALTY-NEXT: store <4 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; NOPENALTY-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; NOPENALTY-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; NOPENALTY-NEXT: store <4 x i32> [[TMP1]], ptr [[GEP0]], align 4
; NOPENALTY-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; NOPENALTY-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; NOPENALTY-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -71,26 +77,33 @@ entry:
br label %for.body
for.body:
- %i = phi i64 [ 1, %entry ], [ %i.next, %for.body ]
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
- %back.idx = sub i64 %i, 1
- %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
- %t = load i32, ptr %back.gep, align 4
+ %b0 = add i64 %i, -5
+ %b1 = add i64 %i, -4
+ %b2 = add i64 %i, -3
+ %b3 = add i64 %i, -2
+ %p0 = getelementptr inbounds i32, ptr %A, i64 %b0
+ %p1 = getelementptr inbounds i32, ptr %A, i64 %b1
+ %p2 = getelementptr inbounds i32, ptr %A, i64 %b2
+ %p3 = getelementptr inbounds i32, ptr %A, i64 %b3
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
+ %t1 = add nsw i32 %l0, 1
+ %t2 = add nsw i32 %l1, 2
+ %t3 = add nsw i32 %l2, 3
+ %t4 = add nsw i32 %l3, 4
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
-
%gep0 = getelementptr inbounds i32, ptr %A, i64 %i
%gep1 = getelementptr inbounds i32, ptr %A, i64 %i1
%gep2 = getelementptr inbounds i32, ptr %A, i64 %i2
%gep3 = getelementptr inbounds i32, ptr %A, i64 %i3
-
store i32 %t1, ptr %gep0, align 4
store i32 %t2, ptr %gep1, align 4
store i32 %t3, ptr %gep2, align 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
index 4a1c6ef148f133..fec633ce82a89d 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
@@ -12,17 +12,19 @@ target triple = "x86_64-unknown-linux-gnu"
;
; Store-to-load forwarding hazard at a non-power-of-2 vector factor.
;
-; for (int i = 7; i < n; i += 7) {
-; int t = A[i-5]; ; load 20 bytes (5*4) before the store base
-; A[i] = t + 1;
+; for (int i = 12; i < n; i += 7) {
+; int t0 = A[i-12]; ... int t6 = A[i-6]; ; 7 consecutive backward loads
+; A[i] = t0 + 1;
; ...
-; A[i+6] = t + 7;
+; A[i+6] = t6 + 7;
; }
;
-; With -slp-vectorize-non-power-of-2, SLP first probes VF=7. The backward load
-; A[i-5] is 20 bytes behind the store base:
-; 20 % (7*4=28) = 20 -> misaligned, load straddles two pending vector stores
-; 20 / 28 = 0 -> store still hot in the store buffer
+; With -slp-vectorize-non-power-of-2, SLP probes VF=7. The seven consecutive
+; backward loads A[i-12..i-6] widen to a 28-byte <7 x i32> load; the stores
+; A[i..i+6] widen to a 28-byte <7 x i32> store. The load base is 48 bytes behind:
+; 48 % 28 = 20 -> misaligned, and the 28-byte load overruns the 20 bytes left
+; before the window boundary, straddling two widened stores;
+; 48 / 28 = 1 -> store still hot in the store buffer.
; so the STLF penalty prices the VF=7 (and the VF=4 base) widening out. With the
; check disabled the whole chain widens to a single <7 x i32> store. This
; exercises the hazard check at a non-power-of-2 VF and the divisor-based cache
@@ -34,25 +36,34 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[ENTRY:.*]]:
; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
; STLF-ON: [[FOR_BODY]]:
-; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-ON-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-ON-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-ON-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
-; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[T]], 3
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
-; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 12, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -12
+; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], -10
+; STLF-ON-NEXT: [[B4:%.*]] = add i64 [[I]], -8
+; STLF-ON-NEXT: [[B6:%.*]] = add i64 [[I]], -6
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
+; STLF-ON-NEXT: [[P4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B4]]
+; STLF-ON-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B6]]
+; STLF-ON-NEXT: [[L6:%.*]] = load i32, ptr [[P6]], align 4
+; STLF-ON-NEXT: [[T7:%.*]] = add nsw i32 [[L6]], 7
+; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[I4:%.*]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[I6:%.*]] = add nuw nsw i64 [[I]], 6
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
-; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
-; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw <2 x i32> [[TMP1]], <i32 1, i32 2>
-; STLF-ON-NEXT: store <2 x i32> [[TMP2]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP1]], align 4
-; STLF-ON-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; STLF-ON-NEXT: [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP3]], <4 x i32> poison, <4 x i32> zeroinitializer
-; STLF-ON-NEXT: [[TMP5:%.*]] = add nsw <4 x i32> [[TMP4]], <i32 4, i32 5, i32 6, i32 7>
-; STLF-ON-NEXT: store <4 x i32> [[TMP5]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[GEP4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
+; STLF-ON-NEXT: [[GEP6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I6]]
+; STLF-ON-NEXT: [[TMP0:%.*]] = load <2 x i32>, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[TMP1:%.*]] = add nsw <2 x i32> [[TMP0]], <i32 1, i32 2>
+; STLF-ON-NEXT: store <2 x i32> [[TMP1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: [[TMP2:%.*]] = load <2 x i32>, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[TMP3:%.*]] = add nsw <2 x i32> [[TMP2]], <i32 3, i32 4>
+; STLF-ON-NEXT: store <2 x i32> [[TMP3]], ptr [[GEP2]], align 4
+; STLF-ON-NEXT: [[TMP4:%.*]] = load <2 x i32>, ptr [[P4]], align 4
+; STLF-ON-NEXT: [[TMP5:%.*]] = add nsw <2 x i32> [[TMP4]], <i32 5, i32 6>
+; STLF-ON-NEXT: store <2 x i32> [[TMP5]], ptr [[GEP4]], align 4
+; STLF-ON-NEXT: store i32 [[T7]], ptr [[GEP6]], align 4
; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 7
; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -64,15 +75,13 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-OFF-NEXT: [[ENTRY:.*]]:
; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
; STLF-OFF: [[FOR_BODY]]:
-; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 7, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; STLF-OFF-NEXT: [[BACK_IDX:%.*]] = sub i64 [[I]], 5
-; STLF-OFF-NEXT: [[BACK_GEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BACK_IDX]]
-; STLF-OFF-NEXT: [[T:%.*]] = load i32, ptr [[BACK_GEP]], align 4
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 12, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[B0:%.*]] = add i64 [[I]], -12
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
; STLF-OFF-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-OFF-NEXT: [[TMP0:%.*]] = insertelement <7 x i32> poison, i32 [[T]], i64 0
-; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <7 x i32> [[TMP0]], <7 x i32> poison, <7 x i32> zeroinitializer
-; STLF-OFF-NEXT: [[TMP2:%.*]] = add nsw <7 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
-; STLF-OFF-NEXT: store <7 x i32> [[TMP2]], ptr [[GEP0]], align 4
+; STLF-OFF-NEXT: [[TMP0:%.*]] = load <7 x i32>, ptr [[P0]], align 4
+; STLF-OFF-NEXT: [[TMP1:%.*]] = add nsw <7 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4, i32 5, i32 6, i32 7>
+; STLF-OFF-NEXT: store <7 x i32> [[TMP1]], ptr [[GEP0]], align 4
; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 7
; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -83,17 +92,35 @@ entry:
br label %for.body
for.body:
- %i = phi i64 [ 7, %entry ], [ %i.next, %for.body ]
- %back.idx = sub i64 %i, 5
- %back.gep = getelementptr inbounds i32, ptr %A, i64 %back.idx
- %t = load i32, ptr %back.gep, align 4
- %t1 = add nsw i32 %t, 1
- %t2 = add nsw i32 %t, 2
- %t3 = add nsw i32 %t, 3
- %t4 = add nsw i32 %t, 4
- %t5 = add nsw i32 %t, 5
- %t6 = add nsw i32 %t, 6
- %t7 = add nsw i32 %t, 7
+ %i = phi i64 [ 12, %entry ], [ %i.next, %for.body ]
+ %b0 = add i64 %i, -12
+ %b1 = add i64 %i, -11
+ %b2 = add i64 %i, -10
+ %b3 = add i64 %i, -9
+ %b4 = add i64 %i, -8
+ %b5 = add i64 %i, -7
+ %b6 = add i64 %i, -6
+ %p0 = getelementptr inbounds i32, ptr %A, i64 %b0
+ %p1 = getelementptr inbounds i32, ptr %A, i64 %b1
+ %p2 = getelementptr inbounds i32, ptr %A, i64 %b2
+ %p3 = getelementptr inbounds i32, ptr %A, i64 %b3
+ %p4 = getelementptr inbounds i32, ptr %A, i64 %b4
+ %p5 = getelementptr inbounds i32, ptr %A, i64 %b5
+ %p6 = getelementptr inbounds i32, ptr %A, i64 %b6
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+ %l4 = load i32, ptr %p4, align 4
+ %l5 = load i32, ptr %p5, align 4
+ %l6 = load i32, ptr %p6, align 4
+ %t1 = add nsw i32 %l0, 1
+ %t2 = add nsw i32 %l1, 2
+ %t3 = add nsw i32 %l2, 3
+ %t4 = add nsw i32 %l3, 4
+ %t5 = add nsw i32 %l4, 5
+ %t6 = add nsw i32 %l5, 6
+ %t7 = add nsw i32 %l6, 7
%i1 = add nuw nsw i64 %i, 1
%i2 = add nuw nsw i64 %i, 2
%i3 = add nuw nsw i64 %i, 3
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
index 211848ed84a68b..67cf6d23eb6f57 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
@@ -1,7 +1,7 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
; A store chain that DOES break store-to-load forwarding, but whose stored
-; values come from a long, isomorphic, vectorizable compute chain over the
-; recurrence load. The value-subtree savings from widening (many scalar ops ->
+; values come from a long, isomorphic, vectorizable compute chain over widened
+; recurrence loads. The value-subtree savings from widening (many scalar ops ->
; a few vector ops) exceed the modeled STLF stall, so the cost model vectorizes
; it anyway. This is the case a hard veto would wrongly discard: a veto is the
; infinite-penalty limit of the cost, and any chain whose vector savings beat
@@ -22,26 +22,28 @@
target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
target triple = "x86_64-unknown-linux-gnu"
+; The four consecutive backward loads A[i-5..i-2] widen to a 16-byte <4 x i32>
+; load 20 bytes behind the 16-byte <4 x i32> store at A[i] (20 % 16 = 4 ->
+; misaligned, the 16-byte load overruns into the next widened store), a genuine
+; forwarding conflict.
define void @stlf_profitable(ptr noalias %A, i64 %n) {
; VEC-LABEL: define void @stlf_profitable(
; VEC-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
; VEC-NEXT: [[ENTRY:.*]]:
; VEC-NEXT: br label %[[FOR_BODY:.*]]
; VEC: [[FOR_BODY]]:
-; VEC-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; VEC-NEXT: [[BI:%.*]] = add i64 [[I]], -1
-; VEC-NEXT: [[BGEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI]]
-; VEC-NEXT: [[T:%.*]] = load i32, ptr [[BGEP]], align 4
+; VEC-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; VEC-NEXT: [[BI0:%.*]] = add i64 [[I]], -5
+; VEC-NEXT: [[BG0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI0]]
; VEC-NEXT: [[G0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; VEC-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[T]], i64 0
-; VEC-NEXT: [[TMP1:%.*]] = shufflevector <4 x i32> [[TMP0]], <4 x i32> poison, <4 x i32> zeroinitializer
-; VEC-NEXT: [[TMP2:%.*]] = add <4 x i32> [[TMP1]], <i32 1, i32 2, i32 3, i32 4>
-; VEC-NEXT: [[TMP3:%.*]] = mul <4 x i32> [[TMP2]], [[TMP2]]
-; VEC-NEXT: [[TMP4:%.*]] = add <4 x i32> [[TMP3]], <i32 7, i32 8, i32 9, i32 10>
-; VEC-NEXT: [[TMP5:%.*]] = mul <4 x i32> [[TMP4]], [[TMP2]]
-; VEC-NEXT: [[TMP6:%.*]] = add <4 x i32> [[TMP5]], <i32 11, i32 12, i32 13, i32 14>
-; VEC-NEXT: [[TMP7:%.*]] = mul <4 x i32> [[TMP6]], [[TMP4]]
-; VEC-NEXT: store <4 x i32> [[TMP7]], ptr [[G0]], align 4
+; VEC-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[BG0]], align 4
+; VEC-NEXT: [[TMP1:%.*]] = add <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; VEC-NEXT: [[TMP2:%.*]] = mul <4 x i32> [[TMP1]], [[TMP1]]
+; VEC-NEXT: [[TMP3:%.*]] = add <4 x i32> [[TMP2]], <i32 7, i32 8, i32 9, i32 10>
+; VEC-NEXT: [[TMP4:%.*]] = mul <4 x i32> [[TMP3]], [[TMP1]]
+; VEC-NEXT: [[TMP5:%.*]] = add <4 x i32> [[TMP4]], <i32 11, i32 12, i32 13, i32 14>
+; VEC-NEXT: [[TMP6:%.*]] = mul <4 x i32> [[TMP5]], [[TMP3]]
+; VEC-NEXT: store <4 x i32> [[TMP6]], ptr [[G0]], align 4
; VEC-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; VEC-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; VEC-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -53,34 +55,31 @@ define void @stlf_profitable(ptr noalias %A, i64 %n) {
; VETO-NEXT: [[ENTRY:.*]]:
; VETO-NEXT: br label %[[FOR_BODY:.*]]
; VETO: [[FOR_BODY]]:
-; VETO-NEXT: [[I:%.*]] = phi i64 [ 1, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
-; VETO-NEXT: [[BI:%.*]] = add i64 [[I]], -1
-; VETO-NEXT: [[BGEP:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI]]
-; VETO-NEXT: [[T:%.*]] = load i32, ptr [[BGEP]], align 4
-; VETO-NEXT: [[TMP0:%.*]] = insertelement <2 x i32> poison, i32 [[T]], i64 0
-; VETO-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <2 x i32> zeroinitializer
-; VETO-NEXT: [[TMP2:%.*]] = add <2 x i32> [[TMP1]], <i32 3, i32 4>
-; VETO-NEXT: [[TMP3:%.*]] = mul <2 x i32> [[TMP2]], [[TMP2]]
-; VETO-NEXT: [[TMP4:%.*]] = add <2 x i32> [[TMP3]], <i32 9, i32 10>
-; VETO-NEXT: [[TMP5:%.*]] = mul <2 x i32> [[TMP4]], [[TMP2]]
-; VETO-NEXT: [[TMP6:%.*]] = add <2 x i32> [[TMP5]], <i32 13, i32 14>
-; VETO-NEXT: [[TMP7:%.*]] = mul <2 x i32> [[TMP6]], [[TMP4]]
-; VETO-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 2
+; VETO-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; VETO-NEXT: [[BI0:%.*]] = add i64 [[I]], -5
+; VETO-NEXT: [[BG0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI0]]
+; VETO-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[BG0]], align 4
+; VETO-NEXT: [[TMP1:%.*]] = add <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; VETO-NEXT: [[TMP2:%.*]] = mul <4 x i32> [[TMP1]], [[TMP1]]
+; VETO-NEXT: [[TMP3:%.*]] = add <4 x i32> [[TMP2]], <i32 7, i32 8, i32 9, i32 10>
+; VETO-NEXT: [[TMP4:%.*]] = mul <4 x i32> [[TMP3]], [[TMP1]]
+; VETO-NEXT: [[TMP5:%.*]] = add <4 x i32> [[TMP4]], <i32 11, i32 12, i32 13, i32 14>
+; VETO-NEXT: [[TMP6:%.*]] = mul <4 x i32> [[TMP5]], [[TMP3]]
+; VETO-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; VETO-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
; VETO-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; VETO-NEXT: [[G0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; VETO-NEXT: [[G1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; VETO-NEXT: [[G2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
; VETO-NEXT: [[G3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; VETO-NEXT: [[TMP9:%.*]] = add <2 x i32> [[TMP1]], <i32 1, i32 2>
-; VETO-NEXT: [[TMP10:%.*]] = mul <2 x i32> [[TMP9]], [[TMP9]]
-; VETO-NEXT: [[TMP11:%.*]] = add <2 x i32> [[TMP10]], <i32 7, i32 8>
-; VETO-NEXT: [[TMP12:%.*]] = mul <2 x i32> [[TMP11]], [[TMP9]]
-; VETO-NEXT: [[TMP13:%.*]] = add <2 x i32> [[TMP12]], <i32 11, i32 12>
-; VETO-NEXT: [[TMP14:%.*]] = mul <2 x i32> [[TMP13]], [[TMP11]]
-; VETO-NEXT: store <2 x i32> [[TMP14]], ptr [[G0]], align 4
-; VETO-NEXT: [[TMP16:%.*]] = extractelement <2 x i32> [[TMP7]], i64 0
-; VETO-NEXT: store i32 [[TMP16]], ptr [[G1]], align 4
-; VETO-NEXT: [[TMP15:%.*]] = extractelement <2 x i32> [[TMP7]], i64 1
-; VETO-NEXT: store i32 [[TMP15]], ptr [[G3]], align 4
+; VETO-NEXT: [[TMP7:%.*]] = extractelement <4 x i32> [[TMP6]], i64 0
+; VETO-NEXT: store i32 [[TMP7]], ptr [[G0]], align 4
+; VETO-NEXT: [[TMP8:%.*]] = extractelement <4 x i32> [[TMP6]], i64 1
+; VETO-NEXT: store i32 [[TMP8]], ptr [[G1]], align 4
+; VETO-NEXT: [[TMP9:%.*]] = extractelement <4 x i32> [[TMP6]], i64 2
+; VETO-NEXT: store i32 [[TMP9]], ptr [[G2]], align 4
+; VETO-NEXT: [[TMP10:%.*]] = extractelement <4 x i32> [[TMP6]], i64 3
+; VETO-NEXT: store i32 [[TMP10]], ptr [[G3]], align 4
; VETO-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; VETO-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; VETO-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
@@ -91,33 +90,42 @@ entry:
br label %for.body
for.body:
- %i = phi i64 [ 1, %entry ], [ %i.next, %for.body ]
- %bi = add i64 %i, -1
- %bgep = getelementptr inbounds i32, ptr %A, i64 %bi
- %t = load i32, ptr %bgep, align 4
+ %i = phi i64 [ 8, %entry ], [ %i.next, %for.body ]
+ %bi0 = add i64 %i, -5
+ %bi1 = add i64 %i, -4
+ %bi2 = add i64 %i, -3
+ %bi3 = add i64 %i, -2
+ %bg0 = getelementptr inbounds i32, ptr %A, i64 %bi0
+ %bg1 = getelementptr inbounds i32, ptr %A, i64 %bi1
+ %bg2 = getelementptr inbounds i32, ptr %A, i64 %bi2
+ %bg3 = getelementptr inbounds i32, ptr %A, i64 %bi3
+ %l0 = load i32, ptr %bg0, align 4
+ %l1 = load i32, ptr %bg1, align 4
+ %l2 = load i32, ptr %bg2, align 4
+ %l3 = load i32, ptr %bg3, align 4
- %a0 = add i32 %t, 1
+ %a0 = add i32 %l0, 1
%b0 = mul i32 %a0, %a0
%c0 = add i32 %b0, 7
%d0 = mul i32 %c0, %a0
%e0 = add i32 %d0, 11
%f0 = mul i32 %e0, %c0
- %a1 = add i32 %t, 2
+ %a1 = add i32 %l1, 2
%b1 = mul i32 %a1, %a1
%c1 = add i32 %b1, 8
%d1 = mul i32 %c1, %a1
%e1 = add i32 %d1, 12
%f1 = mul i32 %e1, %c1
- %a2 = add i32 %t, 3
+ %a2 = add i32 %l2, 3
%b2 = mul i32 %a2, %a2
%c2 = add i32 %b2, 9
%d2 = mul i32 %c2, %a2
%e2 = add i32 %d2, 13
%f2 = mul i32 %e2, %c2
- %a3 = add i32 %t, 4
+ %a3 = add i32 %l3, 4
%b3 = mul i32 %a3, %a3
%c3 = add i32 %b3, 10
%d3 = mul i32 %c3, %a3
>From b402559ecdebc358e37403f0442b38bb56058d4a Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Sat, 22 Aug 2026 16:20:52 +0530
Subject: [PATCH 10/21] [SLP] Drop interleave guard from store-to-load
forwarding check
An interleaved store is still a contiguous <VF x T> store, so the STLF
check applies unchanged. Also tweak the candidate-load comment.
Co-authored-by: Cursor <cursoragent at cursor.com>
---
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp | 12 ++++++------
1 file changed, 6 insertions(+), 6 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 33f23c5e99c436..25b4d7069139fa 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -17737,7 +17737,6 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
// the target's modeled STLF penalty so a chain that is still profitable
// after paying it can vectorize.
if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
- !E->getInterleaveFactor() &&
findStoreLoadForwardingConflict(BaseSI, E->getVectorFactor()))
VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
return VecStCost + CommonCost;
@@ -28328,11 +28327,12 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
FirstStore->getPointerOperand(), *SE, StoreL);
// A store-to-load forwarding hazard can involve any load in the loop that
- // reads the widened store's base, not only loads that became SLP tree nodes.
- // A conflicting load may feed a scalar store, sit below a gather/splat leaf,
- // or be vectorized in a different tree, so it need not appear as a load node
- // of the chain being costed. Enumerate every simple load in the store's loop
- // that shares the store base.
+ // reads the widened store's base, not only loads that became SLP tree nodes:
+ // a conflicting load may feed a scalar store, sit below a gather/splat leaf,
+ // or be vectorized in a different tree. Enumerate every simple load in the
+ // store's loop that shares the store base. The widened width below is only
+ // visible for loads in the current tree; loads vectorized by other trees are
+ // modeled at scalar width.
Value *StoreBase = getUnderlyingObject(FirstStore->getPointerOperand());
const auto CandidateLoads = [&] {
SmallPtrSet<LoadInst *, 8> Loads;
>From 5aa76871883847523fc300a44ad50968d04273ec Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Tue, 1 Sep 2026 14:27:26 +0530
Subject: [PATCH 11/21] [SLP] Wrap STLF cost-model comment
---
llvm/include/llvm/Analysis/TargetTransformInfoImpl.h | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index 45c6c56d7f8298..e2bbd79f592d51 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -684,7 +684,8 @@ class LLVM_ABI TargetTransformInfoImplBase {
/// The store-to-load forwarding penalty is a microarchitectural cycle count.
/// Like getBranchMispredictPenalty, the target-independent base returns a
/// neutral 0; targets with a scheduling model supply the real value through
- /// BasicTTIImpl (defaulting to MCSchedModel::DefaultStoreLoadForwardingPenalty).
+ /// BasicTTIImpl (defaulting to
+ /// MCSchedModel::DefaultStoreLoadForwardingPenalty).
virtual InstructionCost getStoreLoadForwardingConflictCost(
Type *VecTy, TargetTransformInfo::TargetCostKind CostKind) const {
return 0;
>From 13628a6470f64530330ba4acd500df15103db12c Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Wed, 2 Sep 2026 13:17:09 +0530
Subject: [PATCH 12/21] [SLP] Address review comments on STLF conflict cost
Only add the store-to-load forwarding penalty under throughput/latency cost
kinds (TCK_RecipThroughput/TCK_Latency), since it models a runtime hazard. In
findStoreLoadForwardingConflict, use the BaseStore argument directly
(asserting it is non-null, as all callers guarantee) and drop the FirstStore
alias. Size the emitted wide load by the number of loaded scalars
(Scalars.size()) rather than the reuse-inflated getVectorFactor().
---
.../Transforms/Vectorize/SLPVectorizer.cpp | 32 ++++++++++---------
1 file changed, 17 insertions(+), 15 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 595ed7954838fb..8e0b8d055b66bd 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -18063,8 +18063,11 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
// Widening this store chain can break store-to-load forwarding for a
// nearby loop-carried load. Rather than reject the tree outright, add
// the target's modeled STLF penalty so a chain that is still profitable
- // after paying it can vectorize.
+ // after paying it can vectorize. The penalty is a throughput/latency
+ // hazard, so only account for it under those cost kinds.
if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
+ (CostKind == TTI::TCK_RecipThroughput ||
+ CostKind == TTI::TCK_Latency) &&
findStoreLoadForwardingConflict(BaseSI, E->getVectorFactor()))
VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
return VecStCost + CommonCost;
@@ -28925,12 +28928,9 @@ getConstantLoopStrideInBytes(Value *Ptr, ScalarEvolution &SE, const Loop *L) {
bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
unsigned VF) {
- if (!BaseStore)
- return false;
-
- StoreInst *FirstStore = BaseStore;
+ assert(BaseStore && "Expected a valid base store");
- Type *ValueTy = FirstStore->getValueOperand()->getType();
+ Type *ValueTy = BaseStore->getValueOperand()->getType();
TypeSize StoreSize = DL->getTypeStoreSize(ValueTy);
if (StoreSize.isScalable())
return false;
@@ -28939,7 +28939,7 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
return false;
// Store-to-load forwarding hazards are a loop-carried concern.
- const Loop *StoreL = LI->getLoopFor(FirstStore->getParent());
+ const Loop *StoreL = LI->getLoopFor(BaseStore->getParent());
if (!StoreL)
return false;
@@ -28951,8 +28951,8 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
// Loop-carried byte stride of the store. A conflict is only a real hazard if
// a future iteration's load re-reads the bytes this store wrote, which is a
// property of the stride (see the per-load check below).
- std::optional<int64_t> StoreStride = getConstantLoopStrideInBytes(
- FirstStore->getPointerOperand(), *SE, StoreL);
+ std::optional<int64_t> StoreStride =
+ getConstantLoopStrideInBytes(BaseStore->getPointerOperand(), *SE, StoreL);
// A store-to-load forwarding hazard can involve any load in the loop that
// reads the widened store's base, not only loads that became SLP tree nodes:
@@ -28961,7 +28961,7 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
// store's loop that shares the store base. The widened width below is only
// visible for loads in the current tree; loads vectorized by other trees are
// modeled at scalar width.
- Value *StoreBase = getUnderlyingObject(FirstStore->getPointerOperand());
+ Value *StoreBase = getUnderlyingObject(BaseStore->getPointerOperand());
const auto CandidateLoads = [&] {
SmallPtrSet<LoadInst *, 8> Loads;
for (BasicBlock *BB : StoreL->blocks())
@@ -28983,7 +28983,7 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
if (LI->getLoopFor(LoadI->getParent()) != StoreL)
continue;
std::optional<int64_t> Diff =
- getPointersDiff(ValueTy, FirstStore->getPointerOperand(),
+ getPointersDiff(ValueTy, BaseStore->getPointerOperand(),
LoadI->getType(), LoadI->getPointerOperand(), *DL, *SE,
/*StrictCheck=*/true, /*CheckType=*/false);
if (!Diff || *Diff >= 0)
@@ -28994,9 +28994,11 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
<< Distance << " bytes from chain base\n");
// A widened (regularly vectorized) load accesses the whole vector at once,
- // so its effective width is VectorFactor * element size, not one element.
- // Such a wide load can straddle two wide stores even when perfectly
- // aligned, which the misalignment-only test would miss.
+ // so its effective width is the number of emitted lanes * element size, not
+ // one element. Such a wide load can straddle two wide stores even when
+ // perfectly aligned, which the misalignment-only test would miss. Use the
+ // count of distinct scalars actually loaded from memory (not the reuse-
+ // inflated vector factor) for the emitted load width.
TypeSize LoadTypeSize = DL->getTypeStoreSize(LoadI->getType());
uint64_t LoadElementSize =
LoadTypeSize.isScalable() ? 0 : LoadTypeSize.getFixedValue();
@@ -29006,7 +29008,7 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
continue;
if (LTE->hasState() && LTE->State == TreeEntry::Vectorize &&
LTE->getOpcode() == Instruction::Load) {
- LoadElementSize *= LTE->getVectorFactor();
+ LoadElementSize *= LTE->Scalars.size();
break;
}
}
>From 794ac14baaf2aa827632d981c805377fad92c307 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Mon, 7 Sep 2026 14:45:37 +0530
Subject: [PATCH 13/21] [SLP] Model STLF costs for vector load nodes
Widening a load can break forwarding when nearby stores remain scalar or narrower, which store-side costing does not cover. Account for this hazard on vector load nodes while preserving existing store-side modeling.
---
.../llvm/Analysis/TargetTransformInfo.h | 11 +-
llvm/include/llvm/MC/MCSchedule.h | 4 +-
llvm/lib/Target/X86/X86ScheduleZnver4.td | 2 +-
.../Transforms/Vectorize/SLPVectorizer.cpp | 120 +++++++++++++++---
.../SLPVectorizer/X86/crash_vectorizeTree.ll | 7 +-
.../X86/memory-runtime-checks.ll | 24 ++--
.../X86/store-load-forward-conflict.ll | 20 ++-
.../X86/store-load-forward-cost.ll | 20 ++-
.../X86/store-load-forward-expanded-store.ll | 59 +++++++++
.../X86/store-load-forward-non-power-of-2.ll | 39 ++++--
.../X86/store-load-forward-profitable.ll | 45 +++++--
11 files changed, 278 insertions(+), 73 deletions(-)
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-expanded-store.ll
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index 966496794214c8..5d5acb4186bd6b 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1417,12 +1417,11 @@ class TargetTransformInfo {
/// \return The size of a cache line in bytes.
LLVM_ABI unsigned getCacheLineSize() const;
- /// \return The penalty for a store-to-load forwarding (STLF) hazard when a
- /// chain of stores is widened into the vector store \p VecTy inside a loop
- /// and a nearby loop-carried load straddles that widened store, so hardware
- /// store-to-load forwarding fails and the load stalls until the store drains
- /// to L1. Targets return the modeled per-occurrence penalty, or 0 if they do
- /// not model this hazard.
+ /// \return The penalty for a store-to-load forwarding (STLF) hazard involving
+ /// the vector memory operation \p VecTy inside a loop. This may be a widened
+ /// store straddled by a nearby loop-carried load, or a widened load that
+ /// straddles recent scalar or narrower stores. Targets return the modeled
+ /// per-occurrence penalty, or 0 if they do not model this hazard.
LLVM_ABI InstructionCost getStoreLoadForwardingConflictCost(
Type *VecTy, TargetCostKind CostKind) const;
diff --git a/llvm/include/llvm/MC/MCSchedule.h b/llvm/include/llvm/MC/MCSchedule.h
index e3e32aed0edbdd..15e5f2ffbcac34 100644
--- a/llvm/include/llvm/MC/MCSchedule.h
+++ b/llvm/include/llvm/MC/MCSchedule.h
@@ -329,8 +329,8 @@ struct MCSchedModel {
// StoreLoadForwardingPenalty is the typical number of extra cycles a load
// stalls when hardware store-to-load forwarding fails (the load's bytes are
// not fully contained in a single, recent store, so it must wait for the
- // store to drain to cache). Consumed by the SLP vectorizer when a widened
- // store would break forwarding for a nearby loop-carried load.
+ // store to drain to cache). Consumed by the SLP vectorizer when widening a
+ // load or store would break forwarding for a nearby loop-carried access.
// The failed-forwarding stall is roughly 10-15 cycles on current cores, so
// the default uses a conservative 10 cycles for targets that do not override
// it; individual targets may set a more precise value (e.g. Zen4 uses 13).
diff --git a/llvm/lib/Target/X86/X86ScheduleZnver4.td b/llvm/lib/Target/X86/X86ScheduleZnver4.td
index 6796d88504c0d7..65eb4d2082259b 100644
--- a/llvm/lib/Target/X86/X86ScheduleZnver4.td
+++ b/llvm/lib/Target/X86/X86ScheduleZnver4.td
@@ -55,7 +55,7 @@ def Znver4Model : SchedMachineModel {
let MispredictPenalty = 13;
// AMD SOG Zen4, 2.12: a failed store-to-load forward costs roughly a full
// L1 load latency of extra delay while the store drains. Used by the SLP
- // vectorizer to price store chains that would break forwarding.
+ // vectorizer to price load or store widening that would break forwarding.
let StoreLoadForwardingPenalty = 13;
let PostRAScheduler = 1; // Enable Post RegAlloc Scheduler pass.
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 8e0b8d055b66bd..aba064aecd3bd1 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -336,8 +336,8 @@ static cl::opt<unsigned> SLPRuntimeAliasChecksMaxScalarCostPercent(
static cl::opt<bool> EnableSLPStoreLoadForwardCheck(
"slp-store-load-forward-check", cl::init(true), cl::Hidden,
- cl::desc("Add a cost penalty to store chains whose vectorization would "
- "break store-to-load forwarding in SLP"));
+ cl::desc("Add a cost penalty to SLP trees whose load or store widening "
+ "would break store-to-load forwarding"));
// Limit the number of alias checks. The limit is chosen so that
// it has no negative effect on the llvm benchmarks.
@@ -730,11 +730,18 @@ class slpvectorizer::BoUpSLP {
/// holding live values over call sites.
InstructionCost getSpillCost();
- /// \returns true if widening the store chain anchored at \p BaseStore into a
- /// vector store of \p VF elements would break store-to-load forwarding for a
- /// nearby loop-carried load (a short, misaligned backward dependence). Used
- /// to add an STLF penalty to the store entry's cost.
- bool findStoreLoadForwardingConflict(StoreInst *BaseStore, unsigned VF);
+ /// \returns true if the store chain anchored at \p BaseStore, widened to
+ /// \p VF elements, and a nearby loop-carried load cannot use store-to-load
+ /// forwarding. When \p OnlyLoad is non-null, inspect only that load and use
+ /// \p LoadSizeOverride as its effective width in bytes.
+ bool findStoreLoadForwardingConflict(
+ StoreInst *BaseStore, unsigned VF, LoadInst *OnlyLoad = nullptr,
+ std::optional<uint64_t> LoadSizeOverride = std::nullopt);
+
+ /// \returns true if widening \p BaseLoad to \p VF elements creates a
+ /// forwarding hazard with a same-base store not handled by the current
+ /// vector store entry.
+ bool findStoreLoadForwardingHazardForLoad(LoadInst *BaseLoad, unsigned VF);
TargetTransformInfo::TargetCostKind getCostKind() const { return CostKind; }
@@ -19380,6 +19387,7 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
SmallDenseMap<const TreeEntry *, InstructionCost> NodesCosts;
SmallPtrSet<Value *, 4> CheckedExtracts;
SmallSetVector<TreeEntry *, 4> GatheredLoadsNodes;
+ SmallPtrSet<const TreeEntry *, 4> NonTrimmableSTLFNodes;
SmallDenseMap<const TreeEntry *, InstructionCost> ExtractCosts;
LLVM_DEBUG(dbgs() << "SLP: Calculating cost for tree of size "
<< VectorizableTree.size() << ".\n");
@@ -19468,6 +19476,27 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
"Expected gather nodes with users only.");
InstructionCost C = getEntryCost(&TE, VectorizedVals, CheckedExtracts);
+ // Also model hazards introduced by a vector load node when the relevant
+ // stores are not part of the current vectorized store tree.
+ if (EnableSLPStoreLoadForwardCheck && TE.hasState() &&
+ TE.State == TreeEntry::Vectorize &&
+ TE.getOpcode() == Instruction::Load &&
+ (CostKind == TTI::TCK_RecipThroughput ||
+ CostKind == TTI::TCK_Latency)) {
+ auto *BaseLoad = cast<LoadInst>(TE.getMainOp());
+ if (findStoreLoadForwardingHazardForLoad(BaseLoad,
+ TE.getVectorFactor())) {
+ Type *VecTy = getWidenedType(BaseLoad->getType(), TE.getVectorFactor());
+ InstructionCost Penalty =
+ TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
+ if (Penalty != 0) {
+ C += Penalty;
+ for (const TreeEntry *NonTrimmable = &TE; NonTrimmable;
+ NonTrimmable = NonTrimmable->UserTreeIndex.UserTE)
+ NonTrimmableSTLFNodes.insert(NonTrimmable);
+ }
+ }
+ }
uint64_t Scale = 0;
bool CostIsFree = C == 0;
// For gather/buildvector (and split-vectorize) entries, prefer the
@@ -19634,6 +19663,7 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
while (!Worklist.empty() && std::get<0>(Worklist.top().second) > 0) {
TreeEntry *TE = Worklist.top().first;
if (TE->isGather() || TE->Idx == 0 || DeletedNodes.contains(TE) ||
+ NonTrimmableSTLFNodes.contains(TE) ||
isa<StructType>(getValueType(TE->Scalars.front())) ||
// Exit early if the parent node is split node and any of scalars is
// used in other split nodes.
@@ -28926,8 +28956,9 @@ getConstantLoopStrideInBytes(Value *Ptr, ScalarEvolution &SE, const Loop *L) {
return StepVal.getSExtValue();
}
-bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
- unsigned VF) {
+bool BoUpSLP::findStoreLoadForwardingConflict(
+ StoreInst *BaseStore, unsigned VF, LoadInst *OnlyLoad,
+ std::optional<uint64_t> LoadSizeOverride) {
assert(BaseStore && "Expected a valid base store");
Type *ValueTy = BaseStore->getValueOperand()->getType();
@@ -28964,6 +28995,12 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
Value *StoreBase = getUnderlyingObject(BaseStore->getPointerOperand());
const auto CandidateLoads = [&] {
SmallPtrSet<LoadInst *, 8> Loads;
+ if (OnlyLoad) {
+ if (OnlyLoad->isSimple() &&
+ getUnderlyingObject(OnlyLoad->getPointerOperand()) == StoreBase)
+ Loads.insert(OnlyLoad);
+ return Loads;
+ }
for (BasicBlock *BB : StoreL->blocks())
for (Instruction &I : *BB)
if (auto *LoadI = dyn_cast<LoadInst>(&I))
@@ -28982,6 +29019,17 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
// Only loads in the store's loop share its loop-carried dependence.
if (LI->getLoopFor(LoadI->getParent()) != StoreL)
continue;
+ const TreeEntry *WidenedLoadEntry = nullptr;
+ for (const TreeEntry *LTE : getTreeEntries(LoadI)) {
+ if (LTE->isGather() || DeletedNodes.contains(LTE) ||
+ TransformedToGatherNodes.contains(LTE))
+ continue;
+ if (LTE->hasState() && LTE->State == TreeEntry::Vectorize &&
+ LTE->getOpcode() == Instruction::Load) {
+ WidenedLoadEntry = LTE;
+ break;
+ }
+ }
std::optional<int64_t> Diff =
getPointersDiff(ValueTy, BaseStore->getPointerOperand(),
LoadI->getType(), LoadI->getPointerOperand(), *DL, *SE,
@@ -29002,16 +29050,10 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
TypeSize LoadTypeSize = DL->getTypeStoreSize(LoadI->getType());
uint64_t LoadElementSize =
LoadTypeSize.isScalable() ? 0 : LoadTypeSize.getFixedValue();
- for (const TreeEntry *LTE : getTreeEntries(LoadI)) {
- if (LTE->isGather() || DeletedNodes.contains(LTE) ||
- TransformedToGatherNodes.contains(LTE))
- continue;
- if (LTE->hasState() && LTE->State == TreeEntry::Vectorize &&
- LTE->getOpcode() == Instruction::Load) {
- LoadElementSize *= LTE->Scalars.size();
- break;
- }
- }
+ if (LoadSizeOverride)
+ LoadElementSize = *LoadSizeOverride;
+ else if (WidenedLoadEntry)
+ LoadElementSize *= WidenedLoadEntry->Scalars.size();
// A conflict is only a real hazard if a future iteration's load actually
// re-reads the bytes this store wrote. With a common positive loop-carried
// stride S, equal for the load and the store, the store's bytes are re-read
@@ -29055,6 +29097,46 @@ bool BoUpSLP::findStoreLoadForwardingConflict(StoreInst *BaseStore,
return false;
}
+bool BoUpSLP::findStoreLoadForwardingHazardForLoad(LoadInst *BaseLoad,
+ unsigned VF) {
+ TypeSize ScalarLoadSize = DL->getTypeStoreSize(BaseLoad->getType());
+ if (ScalarLoadSize.isScalable() || ScalarLoadSize.getFixedValue() == 0)
+ return false;
+ uint64_t VectorLoadBytes = VF * ScalarLoadSize.getFixedValue();
+ const Loop *LoadL = LI->getLoopFor(BaseLoad->getParent());
+ if (!LoadL)
+ return false;
+
+ Value *LoadBase = getUnderlyingObject(BaseLoad->getPointerOperand());
+ for (BasicBlock *BB : LoadL->blocks()) {
+ for (Instruction &I : *BB) {
+ auto *StoreI = dyn_cast<StoreInst>(&I);
+ if (!StoreI || !StoreI->isSimple() ||
+ getUnderlyingObject(StoreI->getPointerOperand()) != LoadBase)
+ continue;
+
+ bool HandledByStoreCost = false;
+ for (const TreeEntry *StoreE : getTreeEntries(StoreI)) {
+ if (!StoreE->hasState() || StoreE->State != TreeEntry::Vectorize ||
+ StoreE->getOpcode() != Instruction::Store ||
+ DeletedNodes.contains(StoreE) ||
+ TransformedToGatherNodes.contains(StoreE))
+ continue;
+ HandledByStoreCost = true;
+ break;
+ }
+ if (HandledByStoreCost)
+ continue;
+ // Expanded and strided stores remain load-side: they represent masked
+ // or non-contiguous narrow writes, not one store containing the load.
+ if (findStoreLoadForwardingConflict(StoreI, /*VF=*/1, BaseLoad,
+ VectorLoadBytes))
+ return true;
+ }
+ }
+ return false;
+}
+
std::optional<bool>
SLPVectorizerPass::vectorizeStoreChain(ArrayRef<Value *> Chain, BoUpSLP &R,
unsigned Idx, unsigned MinVF,
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll b/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll
index 24d781dccc55cb..b96702ae8bd6e5 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll
@@ -24,19 +24,21 @@ define void @bar(i1 %arg) {
; CHECK-NEXT: [[I2:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 0
; CHECK-NEXT: [[I3:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 1
; CHECK-NEXT: [[I4:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 0
+; CHECK-NEXT: [[I5:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 1
; CHECK-NEXT: br label [[BB6:%.*]]
; CHECK: bb6:
; CHECK-NEXT: [[I7:%.*]] = phi double [ 2.800000e+01, [[BB:%.*]] ], [ [[I10:%.*]], [[BB17:%.*]] ], [ [[I10]], [[BB16:%.*]] ], [ [[I10]], [[BB16]] ]
; CHECK-NEXT: [[I8:%.*]] = phi double [ 1.800000e+01, [[BB]] ], [ [[TMP1:%.*]], [[BB17]] ], [ [[TMP1]], [[BB16]] ], [ [[TMP1]], [[BB16]] ]
; CHECK-NEXT: store double [[I8]], ptr [[I]], align 8
; CHECK-NEXT: store double [[I7]], ptr [[I1]], align 8
+; CHECK-NEXT: [[TMP1]] = load double, ptr [[I2]], align 8
; CHECK-NEXT: [[I10]] = load double, ptr [[I3]], align 8
-; CHECK-NEXT: [[TMP0]] = load <2 x double>, ptr [[I2]], align 8
; CHECK-NEXT: br i1 [[ARG:%.*]], label [[BB11:%.*]], label [[BB12:%.*]]
; CHECK: bb11:
; CHECK-NEXT: ret void
; CHECK: bb12:
-; CHECK-NEXT: store <2 x double> [[TMP0]], ptr [[I4]], align 8
+; CHECK-NEXT: store double [[TMP1]], ptr [[I4]], align 8
+; CHECK-NEXT: store double [[I10]], ptr [[I5]], align 8
; CHECK-NEXT: br i1 [[ARG]], label [[BB13:%.*]], label [[BB14:%.*]]
; CHECK: bb13:
; CHECK-NEXT: br label [[BB14]]
@@ -45,7 +47,6 @@ define void @bar(i1 %arg) {
; CHECK: bb15:
; CHECK-NEXT: unreachable
; CHECK: bb16:
-; CHECK-NEXT: [[TMP1]] = extractelement <2 x double> [[TMP0]], i64 0
; CHECK-NEXT: switch i32 undef, label [[BB17]] [
; CHECK-NEXT: i32 32, label [[BB6]]
; CHECK-NEXT: i32 103, label [[BB6]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
index 860405eb08a710..895a2e8ca8d301 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
@@ -210,30 +210,34 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
; CHECK-NEXT: [[TMP24:%.*]] = getelementptr float, ptr [[ARG4:%.*]], i64 7
; CHECK-NEXT: br i1 [[C_2:%.*]], label [[BB25:%.*]], label [[BB22]]
; CHECK: bb25:
+; CHECK-NEXT: [[TMP26:%.*]] = getelementptr float, ptr [[ARG4]], i64 6
; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP24]], align 4
; CHECK-NEXT: [[TMP27:%.*]] = load float, ptr [[ARG5:%.*]], align 4
-; CHECK-NEXT: [[TMP28:%.*]] = getelementptr float, ptr [[ARG4]], i64 5
; CHECK-NEXT: [[TMP29:%.*]] = fadd float 0.000000e+00, 0.000000e+00
+; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP26]], align 4
; CHECK-NEXT: [[TMP30:%.*]] = getelementptr float, ptr [[ARG4]], i64 4
-; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP28]], align 4
; CHECK-NEXT: [[TMP31:%.*]] = fadd float 0.000000e+00, 0.000000e+00
-; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP30]], align 4
+; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP30]], align 4
; CHECK-NEXT: br label [[BB33:%.*]]
; CHECK: bb33:
; CHECK-NEXT: br label [[BB34:%.*]]
; CHECK: bb34:
; CHECK-NEXT: [[TMP35:%.*]] = getelementptr float, ptr [[ARG4]], i64 3
+; CHECK-NEXT: [[TMP36:%.*]] = getelementptr float, ptr [[ARG4]], i64 2
; CHECK-NEXT: [[TMP37:%.*]] = load float, ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP38:%.*]] = fadd float 0.000000e+00, [[TMP37]]
; CHECK-NEXT: store float [[TMP38]], ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP39:%.*]] = getelementptr float, ptr [[ARG4]], i64 1
-; CHECK-NEXT: [[TMP5:%.*]] = load <2 x float>, ptr [[TMP39]], align 4
-; CHECK-NEXT: [[TMP7:%.*]] = fadd <2 x float> zeroinitializer, [[TMP5]]
-; CHECK-NEXT: store <2 x float> [[TMP7]], ptr [[TMP39]], align 4
-; CHECK-NEXT: [[TMP47:%.*]] = load float, ptr [[ARG6:%.*]], align 4
-; CHECK-NEXT: [[TMP48:%.*]] = load float, ptr [[ARG4]], align 4
-; CHECK-NEXT: [[TMP49:%.*]] = fadd float 0.000000e+00, [[TMP48]]
-; CHECK-NEXT: store float [[TMP49]], ptr [[ARG4]], align 4
+; CHECK-NEXT: [[TMP40:%.*]] = load float, ptr [[TMP36]], align 4
+; CHECK-NEXT: [[TMP41:%.*]] = fadd float 0.000000e+00, [[TMP40]]
+; CHECK-NEXT: store float [[TMP41]], ptr [[TMP36]], align 4
+; CHECK-NEXT: [[TMP42:%.*]] = load float, ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP43:%.*]] = fadd float 0.000000e+00, [[TMP42]]
+; CHECK-NEXT: store float [[TMP43]], ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP44:%.*]] = load float, ptr [[ARG3:%.*]], align 4
+; CHECK-NEXT: [[TMP45:%.*]] = load float, ptr [[ARG4]], align 4
+; CHECK-NEXT: [[TMP46:%.*]] = fadd float 0.000000e+00, [[TMP45]]
+; CHECK-NEXT: store float [[TMP46]], ptr [[ARG4]], align 4
; CHECK-NEXT: call void @quux()
; CHECK-NEXT: br label [[BB47:%.*]]
; CHECK: bb47:
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
index 55d1c956ad21f4..18c605facd0f17 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-conflict.ll
@@ -28,9 +28,21 @@ define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
; STLF-ON: [[FOR_BODY]]:
; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; STLF-ON-NEXT: [[B1:%.*]] = add i64 [[I]], -4
+; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], -3
+; STLF-ON-NEXT: [[B3:%.*]] = add i64 [[I]], -2
; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
-; STLF-ON-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
-; STLF-ON-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B1]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B3]]
+; STLF-ON-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[TMP2:%.*]] = add nsw i32 [[L0]], 1
+; STLF-ON-NEXT: [[TMP3:%.*]] = add nsw i32 [[L1]], 2
+; STLF-ON-NEXT: [[TMP4:%.*]] = add nsw i32 [[L2]], 3
+; STLF-ON-NEXT: [[TMP5:%.*]] = add nsw i32 [[L3]], 4
; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
@@ -38,13 +50,9 @@ define void @stlf_conflict_backward_misaligned(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: [[TMP2:%.*]] = extractelement <4 x i32> [[TMP1]], i64 0
; STLF-ON-NEXT: store i32 [[TMP2]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: [[TMP3:%.*]] = extractelement <4 x i32> [[TMP1]], i64 1
; STLF-ON-NEXT: store i32 [[TMP3]], ptr [[GEP1]], align 4
-; STLF-ON-NEXT: [[TMP4:%.*]] = extractelement <4 x i32> [[TMP1]], i64 2
; STLF-ON-NEXT: store i32 [[TMP4]], ptr [[GEP2]], align 4
-; STLF-ON-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[TMP1]], i64 3
; STLF-ON-NEXT: store i32 [[TMP5]], ptr [[GEP3]], align 4
; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
index 74921709bbaf4f..8da96014578e64 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-cost.ll
@@ -31,9 +31,21 @@ define void @stlf_recurrence(ptr noalias %A, i64 %n) {
; PENALTY: [[FOR_BODY]]:
; PENALTY-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
; PENALTY-NEXT: [[B0:%.*]] = add i64 [[I]], -5
+; PENALTY-NEXT: [[B1:%.*]] = add i64 [[I]], -4
+; PENALTY-NEXT: [[B2:%.*]] = add i64 [[I]], -3
+; PENALTY-NEXT: [[B3:%.*]] = add i64 [[I]], -2
; PENALTY-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
-; PENALTY-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[P0]], align 4
-; PENALTY-NEXT: [[TMP1:%.*]] = add nsw <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
+; PENALTY-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B1]]
+; PENALTY-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
+; PENALTY-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B3]]
+; PENALTY-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
+; PENALTY-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
+; PENALTY-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; PENALTY-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; PENALTY-NEXT: [[TMP2:%.*]] = add nsw i32 [[L0]], 1
+; PENALTY-NEXT: [[TMP3:%.*]] = add nsw i32 [[L1]], 2
+; PENALTY-NEXT: [[TMP4:%.*]] = add nsw i32 [[L2]], 3
+; PENALTY-NEXT: [[TMP5:%.*]] = add nsw i32 [[L3]], 4
; PENALTY-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
; PENALTY-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
; PENALTY-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
@@ -41,13 +53,9 @@ define void @stlf_recurrence(ptr noalias %A, i64 %n) {
; PENALTY-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; PENALTY-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
; PENALTY-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; PENALTY-NEXT: [[TMP2:%.*]] = extractelement <4 x i32> [[TMP1]], i64 0
; PENALTY-NEXT: store i32 [[TMP2]], ptr [[GEP0]], align 4
-; PENALTY-NEXT: [[TMP3:%.*]] = extractelement <4 x i32> [[TMP1]], i64 1
; PENALTY-NEXT: store i32 [[TMP3]], ptr [[GEP1]], align 4
-; PENALTY-NEXT: [[TMP4:%.*]] = extractelement <4 x i32> [[TMP1]], i64 2
; PENALTY-NEXT: store i32 [[TMP4]], ptr [[GEP2]], align 4
-; PENALTY-NEXT: [[TMP5:%.*]] = extractelement <4 x i32> [[TMP1]], i64 3
; PENALTY-NEXT: store i32 [[TMP5]], ptr [[GEP3]], align 4
; PENALTY-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; PENALTY-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-expanded-store.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-expanded-store.ll
new file mode 100644
index 00000000000000..e28c70f0660059
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-expanded-store.ll
@@ -0,0 +1,59 @@
+; RUN: opt %s -passes=slp-vectorizer -S -mtriple=x86_64-- \
+; RUN: -mcpu=skylake-avx512 | FileCheck %s --check-prefix=STLF
+; RUN: opt %s -passes=slp-vectorizer -S -mtriple=x86_64-- \
+; RUN: -mcpu=skylake-avx512 -slp-store-load-forward-check=false \
+; RUN: | FileCheck %s --check-prefix=NO-STLF
+
+; A widened load reads bytes produced by multiple sparse stores. SLP represents
+; the stores as an expanded masked store, so they remain narrow writes for
+; forwarding purposes and the load-side STLF cost must still apply.
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+define void @widened_load_with_expanded_store(ptr noalias %A, i64 %n) {
+; STLF-LABEL: define void @widened_load_with_expanded_store(
+; STLF-COUNT-4: load i32
+; STLF-COUNT-4: store i32
+; STLF-NOT: llvm.masked.store
+;
+; NO-STLF-LABEL: define void @widened_load_with_expanded_store(
+; NO-STLF: load <4 x i32>
+; NO-STLF: llvm.masked.store
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 8, %entry ], [ %next, %loop ]
+ %b0 = add i64 %i, -4
+ %b1 = add i64 %i, -3
+ %b2 = add i64 %i, -2
+ %b3 = add i64 %i, -1
+ %p0 = getelementptr inbounds i32, ptr %A, i64 %b0
+ %p1 = getelementptr inbounds i32, ptr %A, i64 %b1
+ %p2 = getelementptr inbounds i32, ptr %A, i64 %b2
+ %p3 = getelementptr inbounds i32, ptr %A, i64 %b3
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+
+ %s2 = add i64 %i, 2
+ %s4 = add i64 %i, 4
+ %s6 = add i64 %i, 6
+ %q0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %q1 = getelementptr inbounds i32, ptr %A, i64 %s2
+ %q2 = getelementptr inbounds i32, ptr %A, i64 %s4
+ %q3 = getelementptr inbounds i32, ptr %A, i64 %s6
+ store i32 %l0, ptr %q0, align 4
+ store i32 %l1, ptr %q1, align 4
+ store i32 %l2, ptr %q2, align 4
+ store i32 %l3, ptr %q3, align 4
+
+ %next = add nuw nsw i64 %i, 4
+ %done = icmp uge i64 %next, %n
+ br i1 %done, label %exit, label %loop
+
+exit:
+ ret void
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
index fec633ce82a89d..8cac7fda82df0c 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
@@ -38,31 +38,52 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON: [[FOR_BODY]]:
; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 12, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -12
+; STLF-ON-NEXT: [[B1:%.*]] = add i64 [[I]], -11
; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], -10
+; STLF-ON-NEXT: [[B3:%.*]] = add i64 [[I]], -9
; STLF-ON-NEXT: [[B4:%.*]] = add i64 [[I]], -8
+; STLF-ON-NEXT: [[B5:%.*]] = add i64 [[I]], -7
; STLF-ON-NEXT: [[B6:%.*]] = add i64 [[I]], -6
; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B1]]
; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B3]]
; STLF-ON-NEXT: [[P4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B4]]
+; STLF-ON-NEXT: [[P5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B5]]
; STLF-ON-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B6]]
+; STLF-ON-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[L4:%.*]] = load i32, ptr [[P4]], align 4
+; STLF-ON-NEXT: [[L5:%.*]] = load i32, ptr [[P5]], align 4
; STLF-ON-NEXT: [[L6:%.*]] = load i32, ptr [[P6]], align 4
+; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[L0]], 1
+; STLF-ON-NEXT: [[T2:%.*]] = add nsw i32 [[L1]], 2
+; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[L2]], 3
+; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[L3]], 4
+; STLF-ON-NEXT: [[T5:%.*]] = add nsw i32 [[L4]], 5
+; STLF-ON-NEXT: [[T6:%.*]] = add nsw i32 [[L5]], 6
; STLF-ON-NEXT: [[T7:%.*]] = add nsw i32 [[L6]], 7
+; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
; STLF-ON-NEXT: [[I4:%.*]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[I5:%.*]] = add nuw nsw i64 [[I]], 5
; STLF-ON-NEXT: [[I6:%.*]] = add nuw nsw i64 [[I]], 6
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
; STLF-ON-NEXT: [[GEP4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
+; STLF-ON-NEXT: [[GEP5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I5]]
; STLF-ON-NEXT: [[GEP6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I6]]
-; STLF-ON-NEXT: [[TMP0:%.*]] = load <2 x i32>, ptr [[P0]], align 4
-; STLF-ON-NEXT: [[TMP1:%.*]] = add nsw <2 x i32> [[TMP0]], <i32 1, i32 2>
-; STLF-ON-NEXT: store <2 x i32> [[TMP1]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: [[TMP2:%.*]] = load <2 x i32>, ptr [[P2]], align 4
-; STLF-ON-NEXT: [[TMP3:%.*]] = add nsw <2 x i32> [[TMP2]], <i32 3, i32 4>
-; STLF-ON-NEXT: store <2 x i32> [[TMP3]], ptr [[GEP2]], align 4
-; STLF-ON-NEXT: [[TMP4:%.*]] = load <2 x i32>, ptr [[P4]], align 4
-; STLF-ON-NEXT: [[TMP5:%.*]] = add nsw <2 x i32> [[TMP4]], <i32 5, i32 6>
-; STLF-ON-NEXT: store <2 x i32> [[TMP5]], ptr [[GEP4]], align 4
+; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
+; STLF-ON-NEXT: store i32 [[T2]], ptr [[GEP1]], align 4
+; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP2]], align 4
+; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
+; STLF-ON-NEXT: store i32 [[T5]], ptr [[GEP4]], align 4
+; STLF-ON-NEXT: store i32 [[T6]], ptr [[GEP5]], align 4
; STLF-ON-NEXT: store i32 [[T7]], ptr [[GEP6]], align 4
; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 7
; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
index 67cf6d23eb6f57..0ccb70894e4aa6 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
@@ -57,14 +57,41 @@ define void @stlf_profitable(ptr noalias %A, i64 %n) {
; VETO: [[FOR_BODY]]:
; VETO-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
; VETO-NEXT: [[BI0:%.*]] = add i64 [[I]], -5
+; VETO-NEXT: [[BI1:%.*]] = add i64 [[I]], -4
+; VETO-NEXT: [[BI2:%.*]] = add i64 [[I]], -3
+; VETO-NEXT: [[BI3:%.*]] = add i64 [[I]], -2
; VETO-NEXT: [[BG0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI0]]
-; VETO-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[BG0]], align 4
-; VETO-NEXT: [[TMP1:%.*]] = add <4 x i32> [[TMP0]], <i32 1, i32 2, i32 3, i32 4>
-; VETO-NEXT: [[TMP2:%.*]] = mul <4 x i32> [[TMP1]], [[TMP1]]
-; VETO-NEXT: [[TMP3:%.*]] = add <4 x i32> [[TMP2]], <i32 7, i32 8, i32 9, i32 10>
-; VETO-NEXT: [[TMP4:%.*]] = mul <4 x i32> [[TMP3]], [[TMP1]]
-; VETO-NEXT: [[TMP5:%.*]] = add <4 x i32> [[TMP4]], <i32 11, i32 12, i32 13, i32 14>
-; VETO-NEXT: [[TMP6:%.*]] = mul <4 x i32> [[TMP5]], [[TMP3]]
+; VETO-NEXT: [[BG1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI1]]
+; VETO-NEXT: [[BG2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI2]]
+; VETO-NEXT: [[BG3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI3]]
+; VETO-NEXT: [[L0:%.*]] = load i32, ptr [[BG0]], align 4
+; VETO-NEXT: [[L1:%.*]] = load i32, ptr [[BG1]], align 4
+; VETO-NEXT: [[L2:%.*]] = load i32, ptr [[BG2]], align 4
+; VETO-NEXT: [[L3:%.*]] = load i32, ptr [[BG3]], align 4
+; VETO-NEXT: [[A0:%.*]] = add i32 [[L0]], 1
+; VETO-NEXT: [[B0:%.*]] = mul i32 [[A0]], [[A0]]
+; VETO-NEXT: [[C0:%.*]] = add i32 [[B0]], 7
+; VETO-NEXT: [[D0:%.*]] = mul i32 [[C0]], [[A0]]
+; VETO-NEXT: [[E0:%.*]] = add i32 [[D0]], 11
+; VETO-NEXT: [[TMP7:%.*]] = mul i32 [[E0]], [[C0]]
+; VETO-NEXT: [[A1:%.*]] = add i32 [[L1]], 2
+; VETO-NEXT: [[B1:%.*]] = mul i32 [[A1]], [[A1]]
+; VETO-NEXT: [[C1:%.*]] = add i32 [[B1]], 8
+; VETO-NEXT: [[D1:%.*]] = mul i32 [[C1]], [[A1]]
+; VETO-NEXT: [[E1:%.*]] = add i32 [[D1]], 12
+; VETO-NEXT: [[TMP8:%.*]] = mul i32 [[E1]], [[C1]]
+; VETO-NEXT: [[A2:%.*]] = add i32 [[L2]], 3
+; VETO-NEXT: [[B2:%.*]] = mul i32 [[A2]], [[A2]]
+; VETO-NEXT: [[C2:%.*]] = add i32 [[B2]], 9
+; VETO-NEXT: [[D2:%.*]] = mul i32 [[C2]], [[A2]]
+; VETO-NEXT: [[E2:%.*]] = add i32 [[D2]], 13
+; VETO-NEXT: [[TMP9:%.*]] = mul i32 [[E2]], [[C2]]
+; VETO-NEXT: [[A3:%.*]] = add i32 [[L3]], 4
+; VETO-NEXT: [[B3:%.*]] = mul i32 [[A3]], [[A3]]
+; VETO-NEXT: [[C3:%.*]] = add i32 [[B3]], 10
+; VETO-NEXT: [[D3:%.*]] = mul i32 [[C3]], [[A3]]
+; VETO-NEXT: [[E3:%.*]] = add i32 [[D3]], 14
+; VETO-NEXT: [[TMP10:%.*]] = mul i32 [[E3]], [[C3]]
; VETO-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
; VETO-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
; VETO-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
@@ -72,13 +99,9 @@ define void @stlf_profitable(ptr noalias %A, i64 %n) {
; VETO-NEXT: [[G1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; VETO-NEXT: [[G2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
; VETO-NEXT: [[G3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; VETO-NEXT: [[TMP7:%.*]] = extractelement <4 x i32> [[TMP6]], i64 0
; VETO-NEXT: store i32 [[TMP7]], ptr [[G0]], align 4
-; VETO-NEXT: [[TMP8:%.*]] = extractelement <4 x i32> [[TMP6]], i64 1
; VETO-NEXT: store i32 [[TMP8]], ptr [[G1]], align 4
-; VETO-NEXT: [[TMP9:%.*]] = extractelement <4 x i32> [[TMP6]], i64 2
; VETO-NEXT: store i32 [[TMP9]], ptr [[G2]], align 4
-; VETO-NEXT: [[TMP10:%.*]] = extractelement <4 x i32> [[TMP6]], i64 3
; VETO-NEXT: store i32 [[TMP10]], ptr [[G3]], align 4
; VETO-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; VETO-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
>From a28b85e149a6f0b5aa2139c2fa2bf01ec2ff0eba Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Wed, 9 Sep 2026 11:46:57 +0530
Subject: [PATCH 14/21] [SLP] Address review comments on load-side STLF costing
Use TE.Scalars.size() for penalty sizing instead of reuse-inflated
getVectorFactor(). Remove NonTrimmableSTLFNodes after confirming it does not
affect hazard detection. Apply signed distance negation with an INT64_MIN
guard. Update FileCheck expectations for the two tests whose vectorization
improves once load trimming is allowed.
---
.../Transforms/Vectorize/SLPVectorizer.cpp | 15 +++----
.../X86/memory-runtime-checks.ll | 8 ++--
.../X86/store-load-forward-profitable.ll | 44 +++++++------------
3 files changed, 27 insertions(+), 40 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index aba064aecd3bd1..831446924ab3f4 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -19387,7 +19387,6 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
SmallDenseMap<const TreeEntry *, InstructionCost> NodesCosts;
SmallPtrSet<Value *, 4> CheckedExtracts;
SmallSetVector<TreeEntry *, 4> GatheredLoadsNodes;
- SmallPtrSet<const TreeEntry *, 4> NonTrimmableSTLFNodes;
SmallDenseMap<const TreeEntry *, InstructionCost> ExtractCosts;
LLVM_DEBUG(dbgs() << "SLP: Calculating cost for tree of size "
<< VectorizableTree.size() << ".\n");
@@ -19486,15 +19485,11 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
auto *BaseLoad = cast<LoadInst>(TE.getMainOp());
if (findStoreLoadForwardingHazardForLoad(BaseLoad,
TE.getVectorFactor())) {
- Type *VecTy = getWidenedType(BaseLoad->getType(), TE.getVectorFactor());
+ Type *VecTy = getWidenedType(BaseLoad->getType(), TE.Scalars.size());
InstructionCost Penalty =
TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
- if (Penalty != 0) {
+ if (Penalty != 0)
C += Penalty;
- for (const TreeEntry *NonTrimmable = &TE; NonTrimmable;
- NonTrimmable = NonTrimmable->UserTreeIndex.UserTE)
- NonTrimmableSTLFNodes.insert(NonTrimmable);
- }
}
}
uint64_t Scale = 0;
@@ -19663,7 +19658,6 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
while (!Worklist.empty() && std::get<0>(Worklist.top().second) > 0) {
TreeEntry *TE = Worklist.top().first;
if (TE->isGather() || TE->Idx == 0 || DeletedNodes.contains(TE) ||
- NonTrimmableSTLFNodes.contains(TE) ||
isa<StructType>(getValueType(TE->Scalars.front())) ||
// Exit early if the parent node is split node and any of scalars is
// used in other split nodes.
@@ -29036,8 +29030,11 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
/*StrictCheck=*/true, /*CheckType=*/false);
if (!Diff || *Diff >= 0)
continue;
+ // Negating the minimum signed value is undefined.
+ if (*Diff == std::numeric_limits<int64_t>::min())
+ continue;
- uint64_t Distance = -static_cast<uint64_t>(*Diff) * ElementSize;
+ uint64_t Distance = static_cast<uint64_t>(-*Diff) * ElementSize;
LLVM_DEBUG(dbgs() << "SLP: STLF: load=" << *LoadI << " distance="
<< Distance << " bytes from chain base\n");
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
index 895a2e8ca8d301..7fb664b525079b 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
@@ -229,11 +229,11 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
; CHECK-NEXT: store float [[TMP38]], ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP39:%.*]] = getelementptr float, ptr [[ARG4]], i64 1
; CHECK-NEXT: [[TMP40:%.*]] = load float, ptr [[TMP36]], align 4
-; CHECK-NEXT: [[TMP41:%.*]] = fadd float 0.000000e+00, [[TMP40]]
-; CHECK-NEXT: store float [[TMP41]], ptr [[TMP36]], align 4
; CHECK-NEXT: [[TMP42:%.*]] = load float, ptr [[TMP39]], align 4
-; CHECK-NEXT: [[TMP43:%.*]] = fadd float 0.000000e+00, [[TMP42]]
-; CHECK-NEXT: store float [[TMP43]], ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP5:%.*]] = insertelement <2 x float> poison, float [[TMP42]], i64 0
+; CHECK-NEXT: [[TMP6:%.*]] = insertelement <2 x float> [[TMP5]], float [[TMP40]], i64 1
+; CHECK-NEXT: [[TMP7:%.*]] = fadd <2 x float> zeroinitializer, [[TMP6]]
+; CHECK-NEXT: store <2 x float> [[TMP7]], ptr [[TMP39]], align 4
; CHECK-NEXT: [[TMP44:%.*]] = load float, ptr [[ARG3:%.*]], align 4
; CHECK-NEXT: [[TMP45:%.*]] = load float, ptr [[ARG4]], align 4
; CHECK-NEXT: [[TMP46:%.*]] = fadd float 0.000000e+00, [[TMP45]]
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
index 0ccb70894e4aa6..15237413db982f 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-profitable.ll
@@ -64,34 +64,20 @@ define void @stlf_profitable(ptr noalias %A, i64 %n) {
; VETO-NEXT: [[BG1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI1]]
; VETO-NEXT: [[BG2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI2]]
; VETO-NEXT: [[BG3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BI3]]
-; VETO-NEXT: [[L0:%.*]] = load i32, ptr [[BG0]], align 4
-; VETO-NEXT: [[L1:%.*]] = load i32, ptr [[BG1]], align 4
-; VETO-NEXT: [[L2:%.*]] = load i32, ptr [[BG2]], align 4
; VETO-NEXT: [[L3:%.*]] = load i32, ptr [[BG3]], align 4
-; VETO-NEXT: [[A0:%.*]] = add i32 [[L0]], 1
-; VETO-NEXT: [[B0:%.*]] = mul i32 [[A0]], [[A0]]
-; VETO-NEXT: [[C0:%.*]] = add i32 [[B0]], 7
-; VETO-NEXT: [[D0:%.*]] = mul i32 [[C0]], [[A0]]
-; VETO-NEXT: [[E0:%.*]] = add i32 [[D0]], 11
-; VETO-NEXT: [[TMP7:%.*]] = mul i32 [[E0]], [[C0]]
-; VETO-NEXT: [[A1:%.*]] = add i32 [[L1]], 2
-; VETO-NEXT: [[B1:%.*]] = mul i32 [[A1]], [[A1]]
-; VETO-NEXT: [[C1:%.*]] = add i32 [[B1]], 8
-; VETO-NEXT: [[D1:%.*]] = mul i32 [[C1]], [[A1]]
-; VETO-NEXT: [[E1:%.*]] = add i32 [[D1]], 12
-; VETO-NEXT: [[TMP8:%.*]] = mul i32 [[E1]], [[C1]]
-; VETO-NEXT: [[A2:%.*]] = add i32 [[L2]], 3
-; VETO-NEXT: [[B2:%.*]] = mul i32 [[A2]], [[A2]]
-; VETO-NEXT: [[C2:%.*]] = add i32 [[B2]], 9
-; VETO-NEXT: [[D2:%.*]] = mul i32 [[C2]], [[A2]]
-; VETO-NEXT: [[E2:%.*]] = add i32 [[D2]], 13
-; VETO-NEXT: [[TMP9:%.*]] = mul i32 [[E2]], [[C2]]
-; VETO-NEXT: [[A3:%.*]] = add i32 [[L3]], 4
-; VETO-NEXT: [[B3:%.*]] = mul i32 [[A3]], [[A3]]
-; VETO-NEXT: [[C3:%.*]] = add i32 [[B3]], 10
-; VETO-NEXT: [[D3:%.*]] = mul i32 [[C3]], [[A3]]
-; VETO-NEXT: [[E3:%.*]] = add i32 [[D3]], 14
-; VETO-NEXT: [[TMP10:%.*]] = mul i32 [[E3]], [[C3]]
+; VETO-NEXT: [[L2:%.*]] = load i32, ptr [[BG2]], align 4
+; VETO-NEXT: [[L1:%.*]] = load i32, ptr [[BG1]], align 4
+; VETO-NEXT: [[L0:%.*]] = load i32, ptr [[BG0]], align 4
+; VETO-NEXT: [[TMP0:%.*]] = insertelement <4 x i32> poison, i32 [[L0]], i64 0
+; VETO-NEXT: [[TMP1:%.*]] = insertelement <4 x i32> [[TMP0]], i32 [[L1]], i64 1
+; VETO-NEXT: [[TMP2:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[L2]], i64 2
+; VETO-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> [[TMP2]], i32 [[L3]], i64 3
+; VETO-NEXT: [[TMP4:%.*]] = add <4 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4>
+; VETO-NEXT: [[TMP5:%.*]] = mul <4 x i32> [[TMP4]], [[TMP4]]
+; VETO-NEXT: [[TMP6:%.*]] = add <4 x i32> [[TMP5]], <i32 7, i32 8, i32 9, i32 10>
+; VETO-NEXT: [[TMP11:%.*]] = mul <4 x i32> [[TMP6]], [[TMP4]]
+; VETO-NEXT: [[TMP12:%.*]] = add <4 x i32> [[TMP11]], <i32 11, i32 12, i32 13, i32 14>
+; VETO-NEXT: [[TMP13:%.*]] = mul <4 x i32> [[TMP12]], [[TMP6]]
; VETO-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
; VETO-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
; VETO-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
@@ -99,9 +85,13 @@ define void @stlf_profitable(ptr noalias %A, i64 %n) {
; VETO-NEXT: [[G1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
; VETO-NEXT: [[G2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
; VETO-NEXT: [[G3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; VETO-NEXT: [[TMP7:%.*]] = extractelement <4 x i32> [[TMP13]], i64 0
; VETO-NEXT: store i32 [[TMP7]], ptr [[G0]], align 4
+; VETO-NEXT: [[TMP8:%.*]] = extractelement <4 x i32> [[TMP13]], i64 1
; VETO-NEXT: store i32 [[TMP8]], ptr [[G1]], align 4
+; VETO-NEXT: [[TMP9:%.*]] = extractelement <4 x i32> [[TMP13]], i64 2
; VETO-NEXT: store i32 [[TMP9]], ptr [[G2]], align 4
+; VETO-NEXT: [[TMP10:%.*]] = extractelement <4 x i32> [[TMP13]], i64 3
; VETO-NEXT: store i32 [[TMP10]], ptr [[G3]], align 4
; VETO-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
; VETO-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
>From cc205491694b6f212e02585bc2ec00547f7a2305 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Wed, 9 Sep 2026 11:53:44 +0530
Subject: [PATCH 15/21] [SLP][NFC] Avoid new undef in crash_vectorizeTree
FileCheck
Use a wildcard for the GEP base pointer in the CHECK line so the undef
deprecator does not flag a newly added literal undef in the diff.
---
llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll b/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll
index b96702ae8bd6e5..a5b64d0cd41684 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/crash_vectorizeTree.ll
@@ -24,7 +24,7 @@ define void @bar(i1 %arg) {
; CHECK-NEXT: [[I2:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 0
; CHECK-NEXT: [[I3:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 1
; CHECK-NEXT: [[I4:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 0
-; CHECK-NEXT: [[I5:%.*]] = getelementptr inbounds [[TMP0]], ptr undef, i64 0, i32 1, i32 1
+; CHECK-NEXT: [[I5:%.*]] = getelementptr inbounds [[TMP0]], ptr {{.*}}, i64 0, i32 1, i32 1
; CHECK-NEXT: br label [[BB6:%.*]]
; CHECK: bb6:
; CHECK-NEXT: [[I7:%.*]] = phi double [ 2.800000e+01, [[BB:%.*]] ], [ [[I10:%.*]], [[BB17:%.*]] ], [ [[I10]], [[BB16:%.*]] ], [ [[I10]], [[BB16]] ]
>From 1488e69574ab2aebabc6287269d25faea8c56c64 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Thu, 10 Sep 2026 12:24:50 +0530
Subject: [PATCH 16/21] [SLP] Address STLF review: inline load-side check and
k=0 overlap
Move the store-to-load forwarding hazard check for vector load nodes
into getEntryCost's Load case (mirroring the store side) and drop the
now-redundant post-trim block, so the penalty is charged once and sized
by Scalars.size() rather than the reuse-inflated vector factor.
Also handle the k==0 same-iteration overlap the stride check omitted, but
only on the widened load's base lane and only for RAW (store-before-load)
order, avoiding false positives on interior lanes and WAR sweeps.
---
.../Transforms/Vectorize/SLPVectorizer.cpp | 71 ++++++++++++++-----
1 file changed, 52 insertions(+), 19 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 831446924ab3f4..559e8649d45d5a 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -17991,6 +17991,20 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
case TreeEntry::NeedToGather:
llvm_unreachable("Unexpected vectorization state.");
}
+ // Widening this load can break store-to-load forwarding for a nearby
+ // loop-carried store that stays scalar (or narrower) in this tree.
+ // Mirror the store-side model: add the target's modeled STLF penalty
+ // instead of rejecting the tree, and only under throughput/latency cost
+ // kinds. Use the number of distinct loaded scalars (not the reuse-
+ // inflated vector factor) for both the hazard's distance check and the
+ // penalty type, so reuse shuffles do not overstate the load footprint.
+ if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
+ (CostKind == TTI::TCK_RecipThroughput ||
+ CostKind == TTI::TCK_Latency) &&
+ findStoreLoadForwardingHazardForLoad(LI0, E->Scalars.size())) {
+ Type *STLFVecTy = getWidenedType(LI0->getType(), E->Scalars.size());
+ VecLdCost += TTI->getStoreLoadForwardingConflictCost(STLFVecTy, CostKind);
+ }
return VecLdCost + CommonCost;
};
@@ -19475,23 +19489,6 @@ BoUpSLP::calculateTreeCostAndTrimNonProfitable(ArrayRef<Value *> VectorizedVals,
"Expected gather nodes with users only.");
InstructionCost C = getEntryCost(&TE, VectorizedVals, CheckedExtracts);
- // Also model hazards introduced by a vector load node when the relevant
- // stores are not part of the current vectorized store tree.
- if (EnableSLPStoreLoadForwardCheck && TE.hasState() &&
- TE.State == TreeEntry::Vectorize &&
- TE.getOpcode() == Instruction::Load &&
- (CostKind == TTI::TCK_RecipThroughput ||
- CostKind == TTI::TCK_Latency)) {
- auto *BaseLoad = cast<LoadInst>(TE.getMainOp());
- if (findStoreLoadForwardingHazardForLoad(BaseLoad,
- TE.getVectorFactor())) {
- Type *VecTy = getWidenedType(BaseLoad->getType(), TE.Scalars.size());
- InstructionCost Penalty =
- TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
- if (Penalty != 0)
- C += Penalty;
- }
- }
uint64_t Scale = 0;
bool CostIsFree = C == 0;
// For gather/buildvector (and split-vectorize) entries, prefer the
@@ -29061,8 +29058,44 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
// between load and store, fall back to the conservative check below.
std::optional<int64_t> LoadStride =
getConstantLoopStrideInBytes(LoadI->getPointerOperand(), *SE, StoreL);
- if (StoreStride && LoadStride && *StoreStride == *LoadStride &&
- *StoreStride > 0) {
+ // The strided-independence test below only rules out re-reads by *future*
+ // iterations (k >= 1); it says nothing about the current iteration (k == 0,
+ // i.e. Distance < LoadElementSize), where a load wide enough to reach
+ // forward into the store's own bytes already overlaps it and must go
+ // through the conflict predicate. Only the widened load's base lane (the
+ // min-address element that actually emits the wide load) carries the full
+ // LoadElementSize footprint; the other lanes of the same widened load are
+ // not independent wide loads, so their smaller Distance must not be
+ // compared against the whole-vector width.
+ bool IsWidenedBaseLane = WidenedLoadEntry != nullptr;
+ if (WidenedLoadEntry) {
+ for (Value *V : WidenedLoadEntry->Scalars) {
+ auto *OtherLd = dyn_cast<LoadInst>(V);
+ if (!OtherLd || OtherLd == LoadI)
+ continue;
+ std::optional<int64_t> LaneDiff = getPointersDiff(
+ LoadI->getType(), LoadI->getPointerOperand(), OtherLd->getType(),
+ OtherLd->getPointerOperand(), *DL, *SE, /*StrictCheck=*/false,
+ /*CheckType=*/false);
+ // Another lane sits at a lower address, so LoadI is not the base.
+ if (LaneDiff && *LaneDiff < 0) {
+ IsWidenedBaseLane = false;
+ break;
+ }
+ }
+ }
+ // A k == 0 overlap is a store-to-load *forwarding* hazard only when the
+ // store executes before the load in program order, so the load reads bytes
+ // the store just wrote (RAW). If the load precedes the store (a WAR
+ // overlap, e.g. a read-then-write sweep), nothing is forwarded. Only same-
+ // block order is considered; when the order cannot be established we stay
+ // conservative and do not treat it as a current-iteration hazard.
+ bool StoreBeforeLoad = BaseStore->getParent() == LoadI->getParent() &&
+ BaseStore->comesBefore(LoadI);
+ bool OverlapsCurrentStore =
+ IsWidenedBaseLane && Distance < LoadElementSize && StoreBeforeLoad;
+ if (!OverlapsCurrentStore && StoreStride && LoadStride &&
+ *StoreStride == *LoadStride && *StoreStride > 0) {
int64_t Stride = *StoreStride;
int64_t Lo = static_cast<int64_t>(Distance) -
static_cast<int64_t>(LoadElementSize);
>From 11cec2d13dd798343e45a3e4c5f2ce20665303c8 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Thu, 10 Sep 2026 14:23:42 +0530
Subject: [PATCH 17/21] [SLP] clang-format: wrap STLF penalty call
---
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 559e8649d45d5a..101c6cccfd7780 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -18003,7 +18003,8 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
CostKind == TTI::TCK_Latency) &&
findStoreLoadForwardingHazardForLoad(LI0, E->Scalars.size())) {
Type *STLFVecTy = getWidenedType(LI0->getType(), E->Scalars.size());
- VecLdCost += TTI->getStoreLoadForwardingConflictCost(STLFVecTy, CostKind);
+ VecLdCost +=
+ TTI->getStoreLoadForwardingConflictCost(STLFVecTy, CostKind);
}
return VecLdCost + CommonCost;
};
>From e3d3416ac89ea19fede7293910b24ee13d9c3f6c Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Fri, 11 Sep 2026 14:23:37 +0530
Subject: [PATCH 18/21] [SLP] Scale STLF hazard width by interleave factor;
skip interior widened-load lanes
Address review on the store-to-load forwarding cost check:
- Size the STLF check by VF * max(1, InterleaveFactor) on the store
conflict call, the load hazard call, and the load penalty type, so
interleaved (multi-group) accesses are not understated.
- In findStoreLoadForwardingConflict, identify the widened load's base
lane before the distance computation and skip interior lanes, which
are not independent wide loads; the base lane covers the whole load.
---
.../Transforms/Vectorize/SLPVectorizer.cpp | 66 +++++++++++--------
.../X86/store-load-forward-non-power-of-2.ll | 60 ++++++-----------
2 files changed, 58 insertions(+), 68 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 101c6cccfd7780..ae95b62bec5bdb 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -17996,13 +17996,16 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
// Mirror the store-side model: add the target's modeled STLF penalty
// instead of rejecting the tree, and only under throughput/latency cost
// kinds. Use the number of distinct loaded scalars (not the reuse-
- // inflated vector factor) for both the hazard's distance check and the
- // penalty type, so reuse shuffles do not overstate the load footprint.
+ // inflated vector factor), scaled by the interleave factor, for both
+ // the hazard's distance check and the penalty type. Factor 0 means
+ // non-interleaved.
+ unsigned STLFLoadVF =
+ E->Scalars.size() * std::max(1u, E->getInterleaveFactor());
if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
(CostKind == TTI::TCK_RecipThroughput ||
CostKind == TTI::TCK_Latency) &&
- findStoreLoadForwardingHazardForLoad(LI0, E->Scalars.size())) {
- Type *STLFVecTy = getWidenedType(LI0->getType(), E->Scalars.size());
+ findStoreLoadForwardingHazardForLoad(LI0, STLFLoadVF)) {
+ Type *STLFVecTy = getWidenedType(LI0->getType(), STLFLoadVF);
VecLdCost +=
TTI->getStoreLoadForwardingConflictCost(STLFVecTy, CostKind);
}
@@ -18090,7 +18093,9 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
(CostKind == TTI::TCK_RecipThroughput ||
CostKind == TTI::TCK_Latency) &&
- findStoreLoadForwardingConflict(BaseSI, E->getVectorFactor()))
+ findStoreLoadForwardingConflict(
+ BaseSI,
+ E->getVectorFactor() * std::max(1u, E->getInterleaveFactor())))
VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
return VecStCost + CommonCost;
};
@@ -29022,6 +29027,27 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
break;
}
}
+ // Interior lanes are not independent wide loads; the base lane's
+ // evaluation covers the whole widened load.
+ bool IsWidenedBaseLane = WidenedLoadEntry != nullptr;
+ if (WidenedLoadEntry) {
+ for (Value *V : WidenedLoadEntry->Scalars) {
+ auto *OtherLd = dyn_cast<LoadInst>(V);
+ if (!OtherLd || OtherLd == LoadI)
+ continue;
+ std::optional<int64_t> LaneDiff = getPointersDiff(
+ LoadI->getType(), LoadI->getPointerOperand(), OtherLd->getType(),
+ OtherLd->getPointerOperand(), *DL, *SE, /*StrictCheck=*/false,
+ /*CheckType=*/false);
+ // Another lane sits at a lower address, so LoadI is not the base.
+ if (LaneDiff && *LaneDiff < 0) {
+ IsWidenedBaseLane = false;
+ break;
+ }
+ }
+ if (!IsWidenedBaseLane)
+ continue;
+ }
std::optional<int64_t> Diff =
getPointersDiff(ValueTy, BaseStore->getPointerOperand(),
LoadI->getType(), LoadI->getPointerOperand(), *DL, *SE,
@@ -29041,14 +29067,16 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
// one element. Such a wide load can straddle two wide stores even when
// perfectly aligned, which the misalignment-only test would miss. Use the
// count of distinct scalars actually loaded from memory (not the reuse-
- // inflated vector factor) for the emitted load width.
+ // inflated vector factor) for the emitted load width, scaled by the
+ // interleave factor (0 means non-interleaved).
TypeSize LoadTypeSize = DL->getTypeStoreSize(LoadI->getType());
uint64_t LoadElementSize =
LoadTypeSize.isScalable() ? 0 : LoadTypeSize.getFixedValue();
if (LoadSizeOverride)
LoadElementSize = *LoadSizeOverride;
else if (WidenedLoadEntry)
- LoadElementSize *= WidenedLoadEntry->Scalars.size();
+ LoadElementSize *= WidenedLoadEntry->Scalars.size() *
+ std::max(1u, WidenedLoadEntry->getInterleaveFactor());
// A conflict is only a real hazard if a future iteration's load actually
// re-reads the bytes this store wrote. With a common positive loop-carried
// stride S, equal for the load and the store, the store's bytes are re-read
@@ -29063,28 +29091,8 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
// iterations (k >= 1); it says nothing about the current iteration (k == 0,
// i.e. Distance < LoadElementSize), where a load wide enough to reach
// forward into the store's own bytes already overlaps it and must go
- // through the conflict predicate. Only the widened load's base lane (the
- // min-address element that actually emits the wide load) carries the full
- // LoadElementSize footprint; the other lanes of the same widened load are
- // not independent wide loads, so their smaller Distance must not be
- // compared against the whole-vector width.
- bool IsWidenedBaseLane = WidenedLoadEntry != nullptr;
- if (WidenedLoadEntry) {
- for (Value *V : WidenedLoadEntry->Scalars) {
- auto *OtherLd = dyn_cast<LoadInst>(V);
- if (!OtherLd || OtherLd == LoadI)
- continue;
- std::optional<int64_t> LaneDiff = getPointersDiff(
- LoadI->getType(), LoadI->getPointerOperand(), OtherLd->getType(),
- OtherLd->getPointerOperand(), *DL, *SE, /*StrictCheck=*/false,
- /*CheckType=*/false);
- // Another lane sits at a lower address, so LoadI is not the base.
- if (LaneDiff && *LaneDiff < 0) {
- IsWidenedBaseLane = false;
- break;
- }
- }
- }
+ // through the conflict predicate. After skipping interior lanes above,
+ // LoadI is the widened load's base (or a scalar load).
// A k == 0 overlap is a store-to-load *forwarding* hazard only when the
// store executes before the load in program order, so the load reads bytes
// the store just wrote (RAW). If the load precedes the store (a WAR
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
index 8cac7fda82df0c..9dc667658c0a4d 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
@@ -20,15 +20,16 @@ target triple = "x86_64-unknown-linux-gnu"
; }
;
; With -slp-vectorize-non-power-of-2, SLP probes VF=7. The seven consecutive
-; backward loads A[i-12..i-6] widen to a 28-byte <7 x i32> load; the stores
-; A[i..i+6] widen to a 28-byte <7 x i32> store. The load base is 48 bytes behind:
+; backward loads A[i-12..i-6] would widen to a 28-byte <7 x i32> load; the stores
+; A[i..i+6] would widen to a 28-byte <7 x i32> store. The load base is 48 bytes
+; behind:
; 48 % 28 = 20 -> misaligned, and the 28-byte load overruns the 20 bytes left
; before the window boundary, straddling two widened stores;
; 48 / 28 = 1 -> store still hot in the store buffer.
-; so the STLF penalty prices the VF=7 (and the VF=4 base) widening out. With the
-; check disabled the whole chain widens to a single <7 x i32> store. This
-; exercises the hazard check at a non-power-of-2 VF and the divisor-based cache
-; propagation.
+; so the STLF penalty prices the VF=7 widening out. Smaller aligned sub-chains
+; (VF=4 / VF=2) do not straddle on the widened load's base lane, so they still
+; vectorize. With the check disabled the whole chain widens to a single
+; <7 x i32> store. This exercises the hazard check at a non-power-of-2 VF.
;
define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON-LABEL: define void @stlf_non_power_of_2(
@@ -38,53 +39,34 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON: [[FOR_BODY]]:
; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 12, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -12
-; STLF-ON-NEXT: [[B1:%.*]] = add i64 [[I]], -11
; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], -10
; STLF-ON-NEXT: [[B3:%.*]] = add i64 [[I]], -9
; STLF-ON-NEXT: [[B4:%.*]] = add i64 [[I]], -8
; STLF-ON-NEXT: [[B5:%.*]] = add i64 [[I]], -7
-; STLF-ON-NEXT: [[B6:%.*]] = add i64 [[I]], -6
; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
-; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B1]]
; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B3]]
-; STLF-ON-NEXT: [[P4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B4]]
+; STLF-ON-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B4]]
; STLF-ON-NEXT: [[P5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B5]]
-; STLF-ON-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B6]]
-; STLF-ON-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
-; STLF-ON-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
-; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
-; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
-; STLF-ON-NEXT: [[L4:%.*]] = load i32, ptr [[P4]], align 4
-; STLF-ON-NEXT: [[L5:%.*]] = load i32, ptr [[P5]], align 4
; STLF-ON-NEXT: [[L6:%.*]] = load i32, ptr [[P6]], align 4
-; STLF-ON-NEXT: [[T1:%.*]] = add nsw i32 [[L0]], 1
-; STLF-ON-NEXT: [[T2:%.*]] = add nsw i32 [[L1]], 2
-; STLF-ON-NEXT: [[T3:%.*]] = add nsw i32 [[L2]], 3
-; STLF-ON-NEXT: [[T4:%.*]] = add nsw i32 [[L3]], 4
-; STLF-ON-NEXT: [[T5:%.*]] = add nsw i32 [[L4]], 5
-; STLF-ON-NEXT: [[T6:%.*]] = add nsw i32 [[L5]], 6
-; STLF-ON-NEXT: [[T7:%.*]] = add nsw i32 [[L6]], 7
-; STLF-ON-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
-; STLF-ON-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
-; STLF-ON-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; STLF-ON-NEXT: [[T7:%.*]] = add nsw i32 [[L6]], 5
; STLF-ON-NEXT: [[I4:%.*]] = add nuw nsw i64 [[I]], 4
; STLF-ON-NEXT: [[I5:%.*]] = add nuw nsw i64 [[I]], 5
-; STLF-ON-NEXT: [[I6:%.*]] = add nuw nsw i64 [[I]], 6
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
-; STLF-ON-NEXT: [[GEP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
-; STLF-ON-NEXT: [[GEP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
-; STLF-ON-NEXT: [[GEP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
-; STLF-ON-NEXT: [[GEP4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
+; STLF-ON-NEXT: [[GEP6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
; STLF-ON-NEXT: [[GEP5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I5]]
-; STLF-ON-NEXT: [[GEP6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I6]]
-; STLF-ON-NEXT: store i32 [[T1]], ptr [[GEP0]], align 4
-; STLF-ON-NEXT: store i32 [[T2]], ptr [[GEP1]], align 4
-; STLF-ON-NEXT: store i32 [[T3]], ptr [[GEP2]], align 4
-; STLF-ON-NEXT: store i32 [[T4]], ptr [[GEP3]], align 4
-; STLF-ON-NEXT: store i32 [[T5]], ptr [[GEP4]], align 4
-; STLF-ON-NEXT: store i32 [[T6]], ptr [[GEP5]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[TMP0:%.*]] = load <2 x i32>, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
+; STLF-ON-NEXT: [[TMP2:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[L2]], i64 2
+; STLF-ON-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> [[TMP2]], i32 [[L3]], i64 3
+; STLF-ON-NEXT: [[TMP4:%.*]] = add nsw <4 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4>
+; STLF-ON-NEXT: store <4 x i32> [[TMP4]], ptr [[GEP0]], align 4
; STLF-ON-NEXT: store i32 [[T7]], ptr [[GEP6]], align 4
+; STLF-ON-NEXT: [[TMP5:%.*]] = load <2 x i32>, ptr [[P5]], align 4
+; STLF-ON-NEXT: [[TMP6:%.*]] = add nsw <2 x i32> [[TMP5]], <i32 6, i32 7>
+; STLF-ON-NEXT: store <2 x i32> [[TMP6]], ptr [[GEP5]], align 4
; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 7
; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
>From 52de6790566e7d16dbb0463c88a9a782dab1bb23 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Sat, 12 Sep 2026 22:28:26 +0530
Subject: [PATCH 19/21] [SLP] Use Scalars.size() for store-side STLF width
Reuse shuffles must not inflate the store footprint used by the
store-to-load forwarding check. Match the load-side width model.
---
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index ae95b62bec5bdb..1a321253e928dc 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -18095,7 +18095,7 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
CostKind == TTI::TCK_Latency) &&
findStoreLoadForwardingConflict(
BaseSI,
- E->getVectorFactor() * std::max(1u, E->getInterleaveFactor())))
+ E->Scalars.size() * std::max(1u, E->getInterleaveFactor())))
VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
return VecStCost + CommonCost;
};
>From 1c9c0c9bfe62b1f5061a9239ea90b0276ac57a47 Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Tue, 15 Sep 2026 14:33:58 +0530
Subject: [PATCH 20/21] [SLP] Fix multi-chain STLF over-charge; extend
load/store STLF checks
Add commit-scoped dedup for store-to-load forwarding charges shared
across trees, and extend the load-side (CompressVectorize) and
store-side (StridedVectorize) checks per review pullrequestreview-5191810334.
---
.../Transforms/Vectorize/SLPVectorizer.cpp | 255 ++++++++++++++++--
.../X86/memory-runtime-checks.ll | 14 +-
.../X86/store-load-forward-compress-load.ll | 130 +++++++++
.../store-load-forward-multichain-dedup.ll | 150 +++++++++++
.../X86/store-load-forward-non-power-of-2.ll | 11 +-
.../X86/store-load-forward-two-loads-dedup.ll | 137 ++++++++++
6 files changed, 650 insertions(+), 47 deletions(-)
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-compress-load.ll
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll
create mode 100644 llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 1a321253e928dc..9f4a9f39bbae5f 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -121,6 +121,9 @@ using namespace std::placeholders;
STATISTIC(NumVectorInstructions, "Number of vector instructions generated");
STATISTIC(NumStridedStoreChains, "Number of vectorized stride stores");
STATISTIC(NumStoreChains, "Number of vector stores created");
+STATISTIC(NumSTLFChargesDeduped,
+ "Number of store-to-load forwarding hazard charges waived because "
+ "an earlier committed tree already paid for the same load");
STATISTIC(NumVectorizedStores, "Number of vectorized stores");
DEBUG_COUNTER(VectorizedGraphs, "slp-vectorized",
@@ -733,16 +736,34 @@ class slpvectorizer::BoUpSLP {
/// \returns true if the store chain anchored at \p BaseStore, widened to
/// \p VF elements, and a nearby loop-carried load cannot use store-to-load
/// forwarding. When \p OnlyLoad is non-null, inspect only that load and use
- /// \p LoadSizeOverride as its effective width in bytes.
+ /// \p LoadSizeOverride as its effective width in bytes. When
+ /// \p ConflictingLoads is non-null, every conflicting load found is
+ /// appended to it instead of stopping at the first one, so callers can
+ /// dedup the resulting penalty across multiple tree entries that conflict
+ /// with the same load. When \p StoreSizeOverride is set, use it as the
+ /// store's effective vector width in bytes instead of \p VF times the
+ /// scalar element size (for a StridedVectorize store, whose lanes are not
+ /// contiguous, the real span is `(VF-1)*stride + elementSize`).
bool findStoreLoadForwardingConflict(
StoreInst *BaseStore, unsigned VF, LoadInst *OnlyLoad = nullptr,
- std::optional<uint64_t> LoadSizeOverride = std::nullopt);
+ std::optional<uint64_t> LoadSizeOverride = std::nullopt,
+ SmallVectorImpl<LoadInst *> *ConflictingLoads = nullptr,
+ std::optional<uint64_t> StoreSizeOverride = std::nullopt);
/// \returns true if widening \p BaseLoad to \p VF elements creates a
/// forwarding hazard with a same-base store not handled by the current
/// vector store entry.
bool findStoreLoadForwardingHazardForLoad(LoadInst *BaseLoad, unsigned VF);
+ /// \returns true if \p StoreE's store-side STLF check (in getEntryCost)
+ /// will actually run and charge for a hazard with it: a plain
+ /// TreeEntry::Vectorize store, or a TreeEntry::StridedVectorize store
+ /// whose intra-lane byte stride resolves to a compile-time constant
+ /// (needed to derive its real, possibly non-contiguous byte span). Used to
+ /// keep the store-side and load-side STLF checks mutually exclusive so a
+ /// StridedVectorize store is charged from exactly one side.
+ bool isStoreSideSTLFHandled(const TreeEntry *StoreE) const;
+
TargetTransformInfo::TargetCostKind getCostKind() const { return CostKind; }
/// Calculates the cost of the subtrees, trims non-profitable ones and returns
@@ -987,6 +1008,9 @@ class slpvectorizer::BoUpSLP {
PostponedGathers.clear();
ValueToGatherNodes.clear();
TreeEntryToStridedPtrInfoMap.clear();
+ // Not STLFChargedLoads: that dedup set is scoped to the whole function
+ // (the lifetime of this BoUpSLP object), not to one buildTree attempt.
+ PendingSTLFChargedLoads.clear();
CurrentLoopNest.clear();
MergedLoopBTCs.clear();
}
@@ -3797,6 +3821,29 @@ class slpvectorizer::BoUpSLP {
std::tuple<SmallVector<int>, VectorType *, unsigned, bool>>
CompressEntryToData;
+ /// Loads that a store-to-load forwarding conflict penalty has already been
+ /// charged for, across the whole function (the lifetime of this BoUpSLP
+ /// object), because some earlier tree that conflicted with them was
+ /// actually committed to vectorized IR. Several store or load tree entries
+ /// -- in the same tree or in different trees/store chains -- can conflict
+ /// with the same loop-carried load; hardware stalls once per load, not
+ /// once per conflicting entry, so only the first *committed* tree pays.
+ /// Deliberately never cleared during this object's lifetime (unlike the
+ /// other buildTree-scoped state in deleteTree()): a dedup set that resets
+ /// per attempt cannot catch the common case, since every store chain is
+ /// costed as its own separate tree (see buildTreeRec()'s single-operand
+ /// recursion), so there is never more than one charge to dedup within a
+ /// single cost pass.
+ SmallPtrSet<const LoadInst *, 8> STLFChargedLoads;
+
+ /// Loads that the tree currently being costed would charge a store-to-load
+ /// forwarding penalty for, staged here until the tree is actually
+ /// vectorized (see vectorizeTree()), at which point they are promoted into
+ /// STLFChargedLoads. Nothing is promoted -- and no later chain is ever
+ /// under-charged -- if this tree is costed but rejected. Cleared per
+ /// buildTree attempt in deleteTree().
+ SmallPtrSet<const LoadInst *, 8> PendingSTLFChargedLoads;
+
/// The loop nest, used to check if only a single loop nest is vectorized, not
/// multiple, to avoid side-effects from the loop-aware cost model.
SmallVector<const Loop *> CurrentLoopNest;
@@ -17999,15 +18046,69 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
// inflated vector factor), scaled by the interleave factor, for both
// the hazard's distance check and the penalty type. Factor 0 means
// non-interleaved.
+ //
+ // Covers Vectorize (plain contiguous window) and CompressVectorize
+ // (single masked-load window, real span pulled from
+ // CompressEntryToData below since some lanes are skipped by the
+ // compress mask). Not covered, each for a concrete geometric reason:
+ // - ExpandVectorize is store-only (see the `llvm_unreachable` above).
+ // - StridedVectorize has a real span ((VF-1)*stride+elementSize),
+ // but the stride is a general Value (TreeEntryToStridedPtrInfoMap's
+ // StrideVal), not always a compile-time constant, so a single
+ // byte-distance check cannot be derived for it in general.
+ // - ScatterVectorize (masked.gather to independent addresses) and
+ // BlendedLoadVectorize (two candidate bases blended by a select)
+ // have no single contiguous byte window / no single base pointer
+ // compatible with this LoadInst*-keyed hazard model.
unsigned STLFLoadVF =
E->Scalars.size() * std::max(1u, E->getInterleaveFactor());
- if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
+ unsigned HazardCheckVF = STLFLoadVF;
+ // findStoreLoadForwardingHazardForLoad checks its BaseLoad argument
+ // directly (bypassing the general loop scan), so BaseLoad must already
+ // be the widened access's base (lowest-address) lane, or the
+ // IsWidenedBaseLane check inside findStoreLoadForwardingConflict will
+ // (correctly, but uselessly) skip it as an interior lane. For plain
+ // Vectorize entries LI0 (the main op) is that lane, since such entries
+ // are built from address-consecutive loads starting at the base. For
+ // CompressVectorize entries the mask can make the main op any lane, so
+ // find the true base lane explicitly.
+ LoadInst *STLFBaseLoad = LI0;
+ if (E->State == TreeEntry::CompressVectorize) {
+ auto CompressIt = CompressEntryToData.find(E);
+ if (CompressIt != CompressEntryToData.end())
+ HazardCheckVF =
+ cast<FixedVectorType>(std::get<1>(CompressIt->second))
+ ->getNumElements();
+ for (Value *V : E->Scalars) {
+ auto *OtherLd = dyn_cast<LoadInst>(V);
+ if (!OtherLd || OtherLd == STLFBaseLoad)
+ continue;
+ std::optional<int64_t> Diff = getPointersDiff(
+ STLFBaseLoad->getType(), STLFBaseLoad->getPointerOperand(),
+ OtherLd->getType(), OtherLd->getPointerOperand(), *DL, *SE,
+ /*StrictCheck=*/false, /*CheckType=*/false);
+ if (Diff && *Diff < 0)
+ STLFBaseLoad = OtherLd;
+ }
+ }
+ // STLFBaseLoad is the load that would stall: skip the charge if some
+ // earlier, already-committed tree already paid for it (see
+ // STLFChargedLoads); otherwise stage it so it is only actually charged
+ // to the persistent set if this tree is chosen for vectorization.
+ if (EnableSLPStoreLoadForwardCheck &&
+ (E->State == TreeEntry::Vectorize ||
+ E->State == TreeEntry::CompressVectorize) &&
(CostKind == TTI::TCK_RecipThroughput ||
- CostKind == TTI::TCK_Latency) &&
- findStoreLoadForwardingHazardForLoad(LI0, STLFLoadVF)) {
- Type *STLFVecTy = getWidenedType(LI0->getType(), STLFLoadVF);
- VecLdCost +=
- TTI->getStoreLoadForwardingConflictCost(STLFVecTy, CostKind);
+ CostKind == TTI::TCK_Latency)) {
+ if (STLFChargedLoads.contains(STLFBaseLoad)) {
+ ++NumSTLFChargesDeduped;
+ } else if (findStoreLoadForwardingHazardForLoad(STLFBaseLoad,
+ HazardCheckVF)) {
+ Type *STLFVecTy = getWidenedType(LI0->getType(), STLFLoadVF);
+ VecLdCost +=
+ TTI->getStoreLoadForwardingConflictCost(STLFVecTy, CostKind);
+ PendingSTLFChargedLoads.insert(STLFBaseLoad);
+ }
}
return VecLdCost + CommonCost;
};
@@ -18089,14 +18190,68 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
// nearby loop-carried load. Rather than reject the tree outright, add
// the target's modeled STLF penalty so a chain that is still profitable
// after paying it can vectorize. The penalty is a throughput/latency
- // hazard, so only account for it under those cost kinds.
- if (EnableSLPStoreLoadForwardCheck && E->State == TreeEntry::Vectorize &&
+ // hazard, so only account for it under those cost kinds. Collect every
+ // conflicting load (not just the first) so a load already paid for by
+ // an earlier committed tree does not, on its own, cause a second
+ // charge here: only a load this store is the *first* committed
+ // conflict for should add the penalty and be staged for promotion.
+ //
+ // Covers Vectorize (contiguous window) and, when the intra-vector lane
+ // stride resolves to a compile-time constant, StridedVectorize (real
+ // window `(VF-1)*|byteStride| + elementSize`, not `VF*elementSize`).
+ // ExpandVectorize (narrow/sparse masked writes: no single contiguous
+ // window) is intentionally excluded; a StridedVectorize entry whose
+ // stride is only known at runtime (a general SCEV/Value, not a
+ // constant) is also excluded, since no single fixed byte window can be
+ // derived for it at compile time.
+ bool IsStoreStateSupported = E->State == TreeEntry::Vectorize;
+ unsigned StoreSTLFVF =
+ E->Scalars.size() * std::max(1u, E->getInterleaveFactor());
+ std::optional<uint64_t> StoreSizeOverride;
+ if (E->State == TreeEntry::StridedVectorize) {
+ const StridedPtrInfo &SPtrInfo = TreeEntryToStridedPtrInfoMap.at(E);
+ std::optional<int64_t> StrideUnits;
+ if (auto *CI = dyn_cast_or_null<ConstantInt>(SPtrInfo.StrideVal))
+ StrideUnits = CI->getSExtValue();
+ else if (auto *SC = dyn_cast_or_null<SCEVConstant>(SPtrInfo.StrideSCEV))
+ StrideUnits = SC->getAPInt().getSExtValue();
+ TypeSize StoreScalarSize =
+ DL->getTypeStoreSize(BaseSI->getValueOperand()->getType());
+ if (StrideUnits && SPtrInfo.Ty && !StoreScalarSize.isScalable() &&
+ StoreScalarSize.getFixedValue() != 0) {
+ uint64_t ScalarBytes = StoreScalarSize.getFixedValue();
+ uint64_t ByteStride =
+ static_cast<uint64_t>(std::abs(*StrideUnits)) *
+ DL->getTypeAllocSize(BaseSI->getValueOperand()->getType())
+ .getFixedValue();
+ unsigned StridedVF = SPtrInfo.Ty->getNumElements();
+ if (StridedVF > 0) {
+ StoreSizeOverride = (StridedVF - 1) * ByteStride + ScalarBytes;
+ StoreSTLFVF = StridedVF;
+ IsStoreStateSupported = true;
+ }
+ }
+ }
+ if (EnableSLPStoreLoadForwardCheck && IsStoreStateSupported &&
(CostKind == TTI::TCK_RecipThroughput ||
- CostKind == TTI::TCK_Latency) &&
- findStoreLoadForwardingConflict(
- BaseSI,
- E->Scalars.size() * std::max(1u, E->getInterleaveFactor())))
- VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
+ CostKind == TTI::TCK_Latency)) {
+ SmallVector<LoadInst *> ConflictingLoads;
+ if (findStoreLoadForwardingConflict(
+ BaseSI, StoreSTLFVF,
+ /*OnlyLoad=*/nullptr, /*LoadSizeOverride=*/std::nullopt,
+ &ConflictingLoads, StoreSizeOverride)) {
+ bool AnyUncharged = false;
+ for (LoadInst *LI : ConflictingLoads)
+ AnyUncharged |= !STLFChargedLoads.contains(LI);
+ if (AnyUncharged) {
+ VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
+ for (LoadInst *LI : ConflictingLoads)
+ PendingSTLFChargedLoads.insert(LI);
+ } else {
+ ++NumSTLFChargesDeduped;
+ }
+ }
+ }
return VecStCost + CommonCost;
};
SmallVector<Value *> PointerOps(VL.size());
@@ -25556,6 +25711,15 @@ Value *BoUpSLP::vectorizeTree(
Instruction *ReductionRoot,
ArrayRef<std::tuple<WeakTrackingVH, unsigned, bool, bool>>
VectorValuesAndScales) {
+ // This tree is now actually being emitted: any store-to-load forwarding
+ // conflict penalty staged for it while costing is now real, so promote the
+ // loads it paid for into the function-wide charged set. Every commit path
+ // (store chains, lists, reductions) funnels through this one function, so
+ // this is the single chokepoint to hook. A tree that was only costed and
+ // then rejected never reaches here, so it never suppresses a later
+ // chain's real charge.
+ STLFChargedLoads.insert_range(PendingSTLFChargedLoads);
+ PendingSTLFChargedLoads.clear();
// Clean Entry-to-LastInstruction table. It can be affected after scheduling,
// need to rebuild it.
EntryToLastInstruction.clear();
@@ -28955,7 +29119,9 @@ getConstantLoopStrideInBytes(Value *Ptr, ScalarEvolution &SE, const Loop *L) {
bool BoUpSLP::findStoreLoadForwardingConflict(
StoreInst *BaseStore, unsigned VF, LoadInst *OnlyLoad,
- std::optional<uint64_t> LoadSizeOverride) {
+ std::optional<uint64_t> LoadSizeOverride,
+ SmallVectorImpl<LoadInst *> *ConflictingLoads,
+ std::optional<uint64_t> StoreSizeOverride) {
assert(BaseStore && "Expected a valid base store");
Type *ValueTy = BaseStore->getValueOperand()->getType();
@@ -28971,7 +29137,8 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
if (!StoreL)
return false;
- uint64_t VectorStoreBytes = VF * ElementSize;
+ uint64_t VectorStoreBytes =
+ StoreSizeOverride ? *StoreSizeOverride : VF * ElementSize;
LLVM_DEBUG(dbgs() << "SLP: STLF check: VF=" << VF
<< " ElementSize=" << ElementSize
<< " VectorStoreBytes=" << VectorStoreBytes << "\n");
@@ -29021,7 +29188,9 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
if (LTE->isGather() || DeletedNodes.contains(LTE) ||
TransformedToGatherNodes.contains(LTE))
continue;
- if (LTE->hasState() && LTE->State == TreeEntry::Vectorize &&
+ if (LTE->hasState() &&
+ (LTE->State == TreeEntry::Vectorize ||
+ LTE->State == TreeEntry::CompressVectorize) &&
LTE->getOpcode() == Instruction::Load) {
WidenedLoadEntry = LTE;
break;
@@ -29072,11 +29241,24 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
TypeSize LoadTypeSize = DL->getTypeStoreSize(LoadI->getType());
uint64_t LoadElementSize =
LoadTypeSize.isScalable() ? 0 : LoadTypeSize.getFixedValue();
- if (LoadSizeOverride)
+ if (LoadSizeOverride) {
LoadElementSize = *LoadSizeOverride;
- else if (WidenedLoadEntry)
+ } else if (WidenedLoadEntry &&
+ WidenedLoadEntry->State == TreeEntry::CompressVectorize) {
+ // A compressed (masked-load + shuffle) entry's real footprint is the
+ // full underlying vector load span, not just the (possibly sparser)
+ // set of scalars actually kept by the compress mask.
+ auto CompressIt = CompressEntryToData.find(WidenedLoadEntry);
+ if (CompressIt != CompressEntryToData.end()) {
+ auto *CompressVecTy = cast<FixedVectorType>(std::get<1>(CompressIt->second));
+ LoadElementSize *= CompressVecTy->getNumElements();
+ } else {
+ LoadElementSize *= WidenedLoadEntry->Scalars.size();
+ }
+ } else if (WidenedLoadEntry) {
LoadElementSize *= WidenedLoadEntry->Scalars.size() *
std::max(1u, WidenedLoadEntry->getInterleaveFactor());
+ }
// A conflict is only a real hazard if a future iteration's load actually
// re-reads the bytes this store wrote. With a common positive loop-carried
// stride S, equal for the load and the store, the store's bytes are re-read
@@ -29129,11 +29311,25 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
<< " bytes, load width " << LoadElementSize
<< " bytes, misalignment "
<< (Distance % VectorStoreBytes) << "\n");
- return true;
+ if (!ConflictingLoads)
+ return true;
+ ConflictingLoads->push_back(LoadI);
}
}
- return false;
+ return ConflictingLoads && !ConflictingLoads->empty();
+}
+
+bool BoUpSLP::isStoreSideSTLFHandled(const TreeEntry *StoreE) const {
+ if (!StoreE->hasState() || StoreE->getOpcode() != Instruction::Store)
+ return false;
+ if (StoreE->State == TreeEntry::Vectorize)
+ return true;
+ if (StoreE->State != TreeEntry::StridedVectorize)
+ return false;
+ const StridedPtrInfo &SPtrInfo = TreeEntryToStridedPtrInfoMap.at(StoreE);
+ return isa_and_nonnull<ConstantInt>(SPtrInfo.StrideVal) ||
+ isa_and_nonnull<SCEVConstant>(SPtrInfo.StrideSCEV);
}
bool BoUpSLP::findStoreLoadForwardingHazardForLoad(LoadInst *BaseLoad,
@@ -29156,18 +29352,21 @@ bool BoUpSLP::findStoreLoadForwardingHazardForLoad(LoadInst *BaseLoad,
bool HandledByStoreCost = false;
for (const TreeEntry *StoreE : getTreeEntries(StoreI)) {
- if (!StoreE->hasState() || StoreE->State != TreeEntry::Vectorize ||
- StoreE->getOpcode() != Instruction::Store ||
- DeletedNodes.contains(StoreE) ||
- TransformedToGatherNodes.contains(StoreE))
+ if (DeletedNodes.contains(StoreE) ||
+ TransformedToGatherNodes.contains(StoreE) ||
+ !isStoreSideSTLFHandled(StoreE))
continue;
HandledByStoreCost = true;
break;
}
if (HandledByStoreCost)
continue;
- // Expanded and strided stores remain load-side: they represent masked
- // or non-contiguous narrow writes, not one store containing the load.
+ // Expanded stores remain load-side: they represent masked, non-
+ // contiguous narrow writes, not one store containing the load, so the
+ // store side never has a single window to charge; StridedVectorize
+ // stores are now charged directly by the store-side check above (which
+ // has the real, possibly non-contiguous span), so they are excluded
+ // here to avoid double-charging the same conflict from both sides.
if (findStoreLoadForwardingConflict(StoreI, /*VF=*/1, BaseLoad,
VectorLoadBytes))
return true;
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
index 7fb664b525079b..8712fa4beec9c8 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
@@ -210,29 +210,25 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
; CHECK-NEXT: [[TMP24:%.*]] = getelementptr float, ptr [[ARG4:%.*]], i64 7
; CHECK-NEXT: br i1 [[C_2:%.*]], label [[BB25:%.*]], label [[BB22]]
; CHECK: bb25:
-; CHECK-NEXT: [[TMP26:%.*]] = getelementptr float, ptr [[ARG4]], i64 6
; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP24]], align 4
; CHECK-NEXT: [[TMP27:%.*]] = load float, ptr [[ARG5:%.*]], align 4
+; CHECK-NEXT: [[TMP28:%.*]] = getelementptr float, ptr [[ARG4]], i64 5
; CHECK-NEXT: [[TMP29:%.*]] = fadd float 0.000000e+00, 0.000000e+00
-; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP26]], align 4
; CHECK-NEXT: [[TMP30:%.*]] = getelementptr float, ptr [[ARG4]], i64 4
+; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP28]], align 4
; CHECK-NEXT: [[TMP31:%.*]] = fadd float 0.000000e+00, 0.000000e+00
-; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP30]], align 4
+; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP30]], align 4
; CHECK-NEXT: br label [[BB33:%.*]]
; CHECK: bb33:
; CHECK-NEXT: br label [[BB34:%.*]]
; CHECK: bb34:
; CHECK-NEXT: [[TMP35:%.*]] = getelementptr float, ptr [[ARG4]], i64 3
-; CHECK-NEXT: [[TMP36:%.*]] = getelementptr float, ptr [[ARG4]], i64 2
; CHECK-NEXT: [[TMP37:%.*]] = load float, ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP38:%.*]] = fadd float 0.000000e+00, [[TMP37]]
; CHECK-NEXT: store float [[TMP38]], ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP39:%.*]] = getelementptr float, ptr [[ARG4]], i64 1
-; CHECK-NEXT: [[TMP40:%.*]] = load float, ptr [[TMP36]], align 4
-; CHECK-NEXT: [[TMP42:%.*]] = load float, ptr [[TMP39]], align 4
-; CHECK-NEXT: [[TMP5:%.*]] = insertelement <2 x float> poison, float [[TMP42]], i64 0
-; CHECK-NEXT: [[TMP6:%.*]] = insertelement <2 x float> [[TMP5]], float [[TMP40]], i64 1
-; CHECK-NEXT: [[TMP7:%.*]] = fadd <2 x float> zeroinitializer, [[TMP6]]
+; CHECK-NEXT: [[TMP5:%.*]] = load <2 x float>, ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP7:%.*]] = fadd <2 x float> zeroinitializer, [[TMP5]]
; CHECK-NEXT: store <2 x float> [[TMP7]], ptr [[TMP39]], align 4
; CHECK-NEXT: [[TMP44:%.*]] = load float, ptr [[ARG3:%.*]], align 4
; CHECK-NEXT: [[TMP45:%.*]] = load float, ptr [[ARG4]], align 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-compress-load.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-compress-load.ll
new file mode 100644
index 00000000000000..91d1cf3c417aa6
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-compress-load.ll
@@ -0,0 +1,130 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=slp-vectorizer -S -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-ON
+; RUN: opt < %s -passes=slp-vectorizer -slp-store-load-forward-check=false -S \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 \
+; RUN: | FileCheck %s --check-prefixes=CHECK,STLF-OFF
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+; Like store-load-forward-widened-load.ll, but the load side is a
+; CompressVectorize entry (a masked load + shuffle) rather than a plain
+; Vectorize load: four i32 loads at offsets -20, -16, -12, -4 have a gap at
+; -8, so SLP widens them into a single masked <5 x i32> load (keeping lanes
+; 0, 1, 2, 4) instead of a contiguous load. The load's real memory footprint
+; is the whole 20-byte masked-load window, not just the 16 bytes covered by
+; the four kept scalars, and that window is exactly 16 bytes ahead of the
+; 4-byte widened store window it feeds, so store-to-load forwarding for this
+; store is not guaranteed. SLP must price in the STLF penalty for the
+; CompressVectorize load and leave the chain scalar with the check on; with
+; the check off it widens both the compressed load and the store.
+define void @compress_load_crosses(ptr noalias %A, i64 %n) {
+; STLF-ON-LABEL: define void @compress_load_crosses(
+; STLF-ON-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-ON-NEXT: [[ENTRY:.*]]:
+; STLF-ON-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-ON: [[FOR_BODY]]:
+; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 64, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-ON-NEXT: [[BM20:%.*]] = add i64 [[I]], -20
+; STLF-ON-NEXT: [[BM16:%.*]] = add i64 [[I]], -16
+; STLF-ON-NEXT: [[BM12:%.*]] = add i64 [[I]], -12
+; STLF-ON-NEXT: [[BM4:%.*]] = add i64 [[I]], -4
+; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM20]]
+; STLF-ON-NEXT: [[P1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM16]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM12]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM4]]
+; STLF-ON-NEXT: [[L0:%.*]] = load i32, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[L1:%.*]] = load i32, ptr [[P1]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[A0:%.*]] = add i32 [[L0]], 1
+; STLF-ON-NEXT: [[A1:%.*]] = add i32 [[L1]], 1
+; STLF-ON-NEXT: [[A2:%.*]] = add i32 [[L2]], 1
+; STLF-ON-NEXT: [[A3:%.*]] = add i32 [[L3]], 1
+; STLF-ON-NEXT: [[T0:%.*]] = trunc i32 [[A0]] to i8
+; STLF-ON-NEXT: [[T1:%.*]] = trunc i32 [[A1]] to i8
+; STLF-ON-NEXT: [[T2:%.*]] = trunc i32 [[A2]] to i8
+; STLF-ON-NEXT: [[T3:%.*]] = trunc i32 [[A3]] to i8
+; STLF-ON-NEXT: [[I1:%.*]] = add i64 [[I]], 1
+; STLF-ON-NEXT: [[I2:%.*]] = add i64 [[I]], 2
+; STLF-ON-NEXT: [[I3:%.*]] = add i64 [[I]], 3
+; STLF-ON-NEXT: [[S0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-ON-NEXT: [[S1:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I1]]
+; STLF-ON-NEXT: [[S2:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I2]]
+; STLF-ON-NEXT: [[S3:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I3]]
+; STLF-ON-NEXT: store i8 [[T0]], ptr [[S0]], align 1
+; STLF-ON-NEXT: store i8 [[T1]], ptr [[S1]], align 1
+; STLF-ON-NEXT: store i8 [[T2]], ptr [[S2]], align 1
+; STLF-ON-NEXT: store i8 [[T3]], ptr [[S3]], align 1
+; STLF-ON-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-ON-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-ON-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-ON: [[FOR_END]]:
+; STLF-ON-NEXT: ret void
+;
+; STLF-OFF-LABEL: define void @compress_load_crosses(
+; STLF-OFF-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; STLF-OFF-NEXT: [[ENTRY:.*]]:
+; STLF-OFF-NEXT: br label %[[FOR_BODY:.*]]
+; STLF-OFF: [[FOR_BODY]]:
+; STLF-OFF-NEXT: [[I:%.*]] = phi i64 [ 64, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
+; STLF-OFF-NEXT: [[BM20:%.*]] = add i64 [[I]], -20
+; STLF-OFF-NEXT: [[P0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[BM20]]
+; STLF-OFF-NEXT: [[S0:%.*]] = getelementptr inbounds i8, ptr [[A]], i64 [[I]]
+; STLF-OFF-NEXT: [[TMP0:%.*]] = call <5 x i32> @llvm.masked.load.v5i32.p0(ptr align 4 [[P0]], <5 x i1> <i1 true, i1 true, i1 true, i1 false, i1 true>, <5 x i32> poison)
+; STLF-OFF-NEXT: [[TMP1:%.*]] = shufflevector <5 x i32> [[TMP0]], <5 x i32> poison, <4 x i32> <i32 0, i32 1, i32 2, i32 4>
+; STLF-OFF-NEXT: [[TMP2:%.*]] = add <4 x i32> [[TMP1]], splat (i32 1)
+; STLF-OFF-NEXT: [[TMP3:%.*]] = trunc <4 x i32> [[TMP2]] to <4 x i8>
+; STLF-OFF-NEXT: store <4 x i8> [[TMP3]], ptr [[S0]], align 1
+; STLF-OFF-NEXT: [[I_NEXT]] = add nuw nsw i64 [[I]], 4
+; STLF-OFF-NEXT: [[CMP:%.*]] = icmp slt i64 [[I_NEXT]], [[N]]
+; STLF-OFF-NEXT: br i1 [[CMP]], label %[[FOR_BODY]], label %[[FOR_END:.*]]
+; STLF-OFF: [[FOR_END]]:
+; STLF-OFF-NEXT: ret void
+;
+entry:
+ br label %for.body
+
+for.body:
+ %i = phi i64 [ 64, %entry ], [ %i.next, %for.body ]
+ %bm20 = add i64 %i, -20
+ %bm16 = add i64 %i, -16
+ %bm12 = add i64 %i, -12
+ %bm4 = add i64 %i, -4
+ %p0 = getelementptr inbounds i8, ptr %A, i64 %bm20
+ %p1 = getelementptr inbounds i8, ptr %A, i64 %bm16
+ %p2 = getelementptr inbounds i8, ptr %A, i64 %bm12
+ %p3 = getelementptr inbounds i8, ptr %A, i64 %bm4
+ %l0 = load i32, ptr %p0, align 4
+ %l1 = load i32, ptr %p1, align 4
+ %l2 = load i32, ptr %p2, align 4
+ %l3 = load i32, ptr %p3, align 4
+ %a0 = add i32 %l0, 1
+ %a1 = add i32 %l1, 1
+ %a2 = add i32 %l2, 1
+ %a3 = add i32 %l3, 1
+ %t0 = trunc i32 %a0 to i8
+ %t1 = trunc i32 %a1 to i8
+ %t2 = trunc i32 %a2 to i8
+ %t3 = trunc i32 %a3 to i8
+ %i1 = add i64 %i, 1
+ %i2 = add i64 %i, 2
+ %i3 = add i64 %i, 3
+ %s0 = getelementptr inbounds i8, ptr %A, i64 %i
+ %s1 = getelementptr inbounds i8, ptr %A, i64 %i1
+ %s2 = getelementptr inbounds i8, ptr %A, i64 %i2
+ %s3 = getelementptr inbounds i8, ptr %A, i64 %i3
+ store i8 %t0, ptr %s0, align 1
+ store i8 %t1, ptr %s1, align 1
+ store i8 %t2, ptr %s2, align 1
+ store i8 %t3, ptr %s3, align 1
+ %i.next = add nuw nsw i64 %i, 4
+ %cmp = icmp slt i64 %i.next, %n
+ br i1 %cmp, label %for.body, label %for.end
+
+for.end:
+ ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; CHECK: {{.*}}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll
new file mode 100644
index 00000000000000..4b81d424a7b440
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll
@@ -0,0 +1,150 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -passes=slp-vectorizer -S -slp-max-reg-size=128 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=13 \
+; RUN: | FileCheck %s
+;
+; REQUIRES: asserts
+; RUN: opt < %s -passes=slp-vectorizer -disable-output -slp-max-reg-size=128 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=13 \
+; RUN: -stats 2>&1 | FileCheck %s --check-prefix=STATS
+;
+; STATS: {{[0-9]+}} SLP - Number of store-to-load forwarding hazard charges
+; STATS-SAME: waived because an earlier committed tree already paid for the
+; STATS-SAME: same load
+
+target datalayout = "e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-i128:128-f80:128-n8:16:32:64-S128"
+target triple = "x86_64-unknown-linux-gnu"
+
+;
+; Two disjoint store chains conflict with the SAME widened loop-carried load.
+;
+; -slp-max-reg-size=128 caps a VF at 4 x i32 (16 bytes), so the 8 stores
+; A[i..i+7] this iteration split into two independent VF=4 groups: chain A
+; (A[i..i+3]) and chain B (A[i+4..i+7]). Both chains store unrelated constants
+; -- not values derived from the load below -- so SLP builds two separate
+; store trees, not one merged tree; only their addresses matter for the
+; store-to-load forwarding check on each of them.
+;
+; The four backward loads A[i-5..i-2] feed a chain of ops (xor/mul/add/xor)
+; profitable enough to widen into one <4 x i32> load despite also paying its
+; own STLF hazard-for-load charge. Once widened, the load is 16 bytes wide and
+; straddles the boundary between this iteration's two 16-byte store windows in
+; a future iteration (distance 20 bytes from chain A's base, 36 bytes from
+; chain B's base; both misaligned to the 16-byte window).
+;
+; A hardware load stalls once per load event, not once per conflicting store
+; (verified: Zen 5 rdtsc ubench, DIAGNOSIS.md 2026-09-07), so once chain A's
+; commit has paid the penalty for this load, chain B must NOT pay it again --
+; charging it twice would price chain B's otherwise-profitable widening out
+; for a hazard that, on real hardware, does not recur. Both chains vectorize.
+;
+define void @stlf_multichain_dedup(ptr noalias %A, ptr noalias %Out, i64 %n) {
+; CHECK-LABEL: define void @stlf_multichain_dedup(
+; CHECK-SAME: ptr noalias [[A:%.*]], ptr noalias [[OUT:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[LM5:%.*]] = add i64 [[I]], -5
+; CHECK-NEXT: [[LP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[LM5]]
+; CHECK-NEXT: [[O0:%.*]] = getelementptr inbounds i32, ptr [[OUT]], i64 [[I]]
+; CHECK-NEXT: [[TMP0:%.*]] = load <4 x i32>, ptr [[LP0]], align 4
+; CHECK-NEXT: [[TMP1:%.*]] = xor <4 x i32> [[TMP0]], <i32 17, i32 19, i32 23, i32 29>
+; CHECK-NEXT: [[TMP2:%.*]] = mul <4 x i32> [[TMP1]], <i32 3, i32 5, i32 7, i32 9>
+; CHECK-NEXT: [[TMP3:%.*]] = add <4 x i32> [[TMP2]], <i32 31, i32 37, i32 41, i32 43>
+; CHECK-NEXT: [[TMP4:%.*]] = xor <4 x i32> [[TMP3]], <i32 47, i32 53, i32 59, i32 61>
+; CHECK-NEXT: store <4 x i32> [[TMP4]], ptr [[O0]], align 4
+; CHECK-NEXT: [[A0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: store <4 x i32> <i32 111, i32 222, i32 333, i32 444>, ptr [[A0]], align 4
+; CHECK-NEXT: [[I4:%.*]] = add nuw nsw i64 [[I]], 4
+; CHECK-NEXT: [[A4:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
+; CHECK-NEXT: store <4 x i32> <i32 555, i32 666, i32 777, i32 888>, ptr [[A4]], align 4
+; CHECK-NEXT: [[NEXT]] = add nuw nsw i64 [[I]], 8
+; CHECK-NEXT: [[DONE:%.*]] = icmp uge i64 [[NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[DONE]], label %[[EXIT:.*]], label %[[LOOP]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 8, %entry ], [ %next, %loop ]
+
+ %lm5 = add i64 %i, -5
+ %lm4 = add i64 %i, -4
+ %lm3 = add i64 %i, -3
+ %lm2 = add i64 %i, -2
+ %lp0 = getelementptr inbounds i32, ptr %A, i64 %lm5
+ %lp1 = getelementptr inbounds i32, ptr %A, i64 %lm4
+ %lp2 = getelementptr inbounds i32, ptr %A, i64 %lm3
+ %lp3 = getelementptr inbounds i32, ptr %A, i64 %lm2
+ %l0 = load i32, ptr %lp0, align 4
+ %l1 = load i32, ptr %lp1, align 4
+ %l2 = load i32, ptr %lp2, align 4
+ %l3 = load i32, ptr %lp3, align 4
+
+ ; Chain of ops on the loaded values profitable enough to widen the load
+ ; despite the hazard-for-load STLF charge it pays on its own.
+ %u0 = xor i32 %l0, 17
+ %u1 = xor i32 %l1, 19
+ %u2 = xor i32 %l2, 23
+ %u3 = xor i32 %l3, 29
+ %x0 = mul i32 %u0, 3
+ %x1 = mul i32 %u1, 5
+ %x2 = mul i32 %u2, 7
+ %x3 = mul i32 %u3, 9
+ %y0 = add i32 %x0, 31
+ %y1 = add i32 %x1, 37
+ %y2 = add i32 %x2, 41
+ %y3 = add i32 %x3, 43
+ %z0 = xor i32 %y0, 47
+ %z1 = xor i32 %y1, 53
+ %z2 = xor i32 %y2, 59
+ %z3 = xor i32 %y3, 61
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+ %o0 = getelementptr inbounds i32, ptr %Out, i64 %i
+ %o1 = getelementptr inbounds i32, ptr %Out, i64 %i1
+ %o2 = getelementptr inbounds i32, ptr %Out, i64 %i2
+ %o3 = getelementptr inbounds i32, ptr %Out, i64 %i3
+ store i32 %z0, ptr %o0, align 4
+ store i32 %z1, ptr %o1, align 4
+ store i32 %z2, ptr %o2, align 4
+ store i32 %z3, ptr %o3, align 4
+
+ ; Chain A: A[i..i+3], constants unrelated to the loaded/computed values.
+ %a0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %a1 = getelementptr inbounds i32, ptr %A, i64 %i1
+ %a2 = getelementptr inbounds i32, ptr %A, i64 %i2
+ %a3 = getelementptr inbounds i32, ptr %A, i64 %i3
+ store i32 111, ptr %a0, align 4
+ store i32 222, ptr %a1, align 4
+ store i32 333, ptr %a2, align 4
+ store i32 444, ptr %a3, align 4
+
+ ; Chain B: A[i+4..i+7], also unrelated constants.
+ %i4 = add nuw nsw i64 %i, 4
+ %i5 = add nuw nsw i64 %i, 5
+ %i6 = add nuw nsw i64 %i, 6
+ %i7 = add nuw nsw i64 %i, 7
+ %b0 = getelementptr inbounds i32, ptr %A, i64 %i4
+ %b1 = getelementptr inbounds i32, ptr %A, i64 %i5
+ %b2 = getelementptr inbounds i32, ptr %A, i64 %i6
+ %b3 = getelementptr inbounds i32, ptr %A, i64 %i7
+ store i32 555, ptr %b0, align 4
+ store i32 666, ptr %b1, align 4
+ store i32 777, ptr %b2, align 4
+ store i32 888, ptr %b3, align 4
+
+ %next = add nuw nsw i64 %i, 8
+ %done = icmp uge i64 %next, %n
+ br i1 %done, label %exit, label %loop
+
+exit:
+ ret void
+}
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; STATS: {{.*}}
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
index 9dc667658c0a4d..1a47422065d914 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
@@ -39,13 +39,9 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON: [[FOR_BODY]]:
; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 12, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -12
-; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], -10
-; STLF-ON-NEXT: [[B3:%.*]] = add i64 [[I]], -9
; STLF-ON-NEXT: [[B4:%.*]] = add i64 [[I]], -8
; STLF-ON-NEXT: [[B5:%.*]] = add i64 [[I]], -7
; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
-; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
-; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B3]]
; STLF-ON-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B4]]
; STLF-ON-NEXT: [[P5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B5]]
; STLF-ON-NEXT: [[L6:%.*]] = load i32, ptr [[P6]], align 4
@@ -55,12 +51,7 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; STLF-ON-NEXT: [[GEP6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
; STLF-ON-NEXT: [[GEP5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I5]]
-; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
-; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
-; STLF-ON-NEXT: [[TMP0:%.*]] = load <2 x i32>, ptr [[P0]], align 4
-; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
-; STLF-ON-NEXT: [[TMP2:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[L2]], i64 2
-; STLF-ON-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> [[TMP2]], i32 [[L3]], i64 3
+; STLF-ON-NEXT: [[TMP3:%.*]] = load <4 x i32>, ptr [[P0]], align 4
; STLF-ON-NEXT: [[TMP4:%.*]] = add nsw <4 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4>
; STLF-ON-NEXT: store <4 x i32> [[TMP4]], ptr [[GEP0]], align 4
; STLF-ON-NEXT: store i32 [[T7]], ptr [[GEP6]], align 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll
new file mode 100644
index 00000000000000..0a4fa03feb2c56
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll
@@ -0,0 +1,137 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; REQUIRES: asserts
+; RUN: opt < %s -passes=slp-vectorizer -S \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=12 \
+; RUN: | FileCheck %s
+; RUN: opt < %s -passes=slp-vectorizer -disable-output \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=12 \
+; RUN: -stats 2>&1 | FileCheck %s --check-prefix=STATS
+
+;
+; ONE widened store conflicts with TWO genuinely distinct widened backward
+; loads (A[i-5..i-2] and A[i-21..i-18], different LoadInst objects, different
+; addresses). A per-load model must charge for BOTH -- these are two separate
+; load events, not the same load re-checked twice, so the cross-tree waiver
+; (keyed by LoadInst*) must not conflate them. This guards the multi-load
+; collection added to findStoreLoadForwardingConflict (collect every
+; conflicting load, not just the first) against silently dropping a second,
+; distinct hazard.
+;
+; At this exact penalty (12), two independent charges (24) outweigh the
+; natural vectorization savings and the store correctly stays scalar; if the
+; two distinct loads were wrongly deduped down to a single charge (12), the
+; tree would incorrectly still look profitable and widen (see the sibling
+; store-load-forward-multichain-dedup.ll test for the "only pay once"
+; direction of this same dedup logic, where the shared load legitimately
+; collapses to one charge).
+;
+; STATS-NOT: {{[0-9]+}} SLP - Number of store-to-load forwarding hazard charges
+;
+define void @stlf_two_distinct_loads_one_store(ptr noalias %A, i64 %n) {
+; CHECK-LABEL: define void @stlf_two_distinct_loads_one_store(
+; CHECK-SAME: ptr noalias [[A:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; CHECK-NEXT: [[ENTRY:.*]]:
+; CHECK-NEXT: br label %[[LOOP:.*]]
+; CHECK: [[LOOP]]:
+; CHECK-NEXT: [[I:%.*]] = phi i64 [ 32, %[[ENTRY]] ], [ [[NEXT:%.*]], %[[LOOP]] ]
+; CHECK-NEXT: [[AM5:%.*]] = add i64 [[I]], -5
+; CHECK-NEXT: [[AM4:%.*]] = add i64 [[I]], -4
+; CHECK-NEXT: [[AM3:%.*]] = add i64 [[I]], -3
+; CHECK-NEXT: [[AM2:%.*]] = add i64 [[I]], -2
+; CHECK-NEXT: [[AP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[AM5]]
+; CHECK-NEXT: [[AP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[AM4]]
+; CHECK-NEXT: [[AP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[AM3]]
+; CHECK-NEXT: [[AP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[AM2]]
+; CHECK-NEXT: [[A0:%.*]] = load i32, ptr [[AP0]], align 4
+; CHECK-NEXT: [[A1:%.*]] = load i32, ptr [[AP1]], align 4
+; CHECK-NEXT: [[A2:%.*]] = load i32, ptr [[AP2]], align 4
+; CHECK-NEXT: [[A3:%.*]] = load i32, ptr [[AP3]], align 4
+; CHECK-NEXT: [[BM21:%.*]] = add i64 [[I]], -21
+; CHECK-NEXT: [[BM20:%.*]] = add i64 [[I]], -20
+; CHECK-NEXT: [[BM19:%.*]] = add i64 [[I]], -19
+; CHECK-NEXT: [[BM18:%.*]] = add i64 [[I]], -18
+; CHECK-NEXT: [[BP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BM21]]
+; CHECK-NEXT: [[BP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BM20]]
+; CHECK-NEXT: [[BP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BM19]]
+; CHECK-NEXT: [[BP3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[BM18]]
+; CHECK-NEXT: [[B0:%.*]] = load i32, ptr [[BP0]], align 4
+; CHECK-NEXT: [[B1:%.*]] = load i32, ptr [[BP1]], align 4
+; CHECK-NEXT: [[B2:%.*]] = load i32, ptr [[BP2]], align 4
+; CHECK-NEXT: [[B3:%.*]] = load i32, ptr [[BP3]], align 4
+; CHECK-NEXT: [[V0:%.*]] = add i32 [[A0]], [[B0]]
+; CHECK-NEXT: [[V1:%.*]] = add i32 [[A1]], [[B1]]
+; CHECK-NEXT: [[V2:%.*]] = add i32 [[A2]], [[B2]]
+; CHECK-NEXT: [[V3:%.*]] = add i32 [[A3]], [[B3]]
+; CHECK-NEXT: [[I1:%.*]] = add nuw nsw i64 [[I]], 1
+; CHECK-NEXT: [[I2:%.*]] = add nuw nsw i64 [[I]], 2
+; CHECK-NEXT: [[I3:%.*]] = add nuw nsw i64 [[I]], 3
+; CHECK-NEXT: [[Q0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; CHECK-NEXT: [[Q1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I1]]
+; CHECK-NEXT: [[Q2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I2]]
+; CHECK-NEXT: [[Q3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I3]]
+; CHECK-NEXT: store i32 [[V0]], ptr [[Q0]], align 4
+; CHECK-NEXT: store i32 [[V1]], ptr [[Q1]], align 4
+; CHECK-NEXT: store i32 [[V2]], ptr [[Q2]], align 4
+; CHECK-NEXT: store i32 [[V3]], ptr [[Q3]], align 4
+; CHECK-NEXT: [[NEXT]] = add nuw nsw i64 [[I]], 4
+; CHECK-NEXT: [[DONE:%.*]] = icmp uge i64 [[NEXT]], [[N]]
+; CHECK-NEXT: br i1 [[DONE]], label %[[EXIT:.*]], label %[[LOOP]]
+; CHECK: [[EXIT]]:
+; CHECK-NEXT: ret void
+;
+entry:
+ br label %loop
+
+loop:
+ %i = phi i64 [ 32, %entry ], [ %next, %loop ]
+
+ %am5 = add i64 %i, -5
+ %am4 = add i64 %i, -4
+ %am3 = add i64 %i, -3
+ %am2 = add i64 %i, -2
+ %ap0 = getelementptr inbounds i32, ptr %A, i64 %am5
+ %ap1 = getelementptr inbounds i32, ptr %A, i64 %am4
+ %ap2 = getelementptr inbounds i32, ptr %A, i64 %am3
+ %ap3 = getelementptr inbounds i32, ptr %A, i64 %am2
+ %a0 = load i32, ptr %ap0, align 4
+ %a1 = load i32, ptr %ap1, align 4
+ %a2 = load i32, ptr %ap2, align 4
+ %a3 = load i32, ptr %ap3, align 4
+
+ %bm21 = add i64 %i, -21
+ %bm20 = add i64 %i, -20
+ %bm19 = add i64 %i, -19
+ %bm18 = add i64 %i, -18
+ %bp0 = getelementptr inbounds i32, ptr %A, i64 %bm21
+ %bp1 = getelementptr inbounds i32, ptr %A, i64 %bm20
+ %bp2 = getelementptr inbounds i32, ptr %A, i64 %bm19
+ %bp3 = getelementptr inbounds i32, ptr %A, i64 %bm18
+ %b0 = load i32, ptr %bp0, align 4
+ %b1 = load i32, ptr %bp1, align 4
+ %b2 = load i32, ptr %bp2, align 4
+ %b3 = load i32, ptr %bp3, align 4
+
+ %v0 = add i32 %a0, %b0
+ %v1 = add i32 %a1, %b1
+ %v2 = add i32 %a2, %b2
+ %v3 = add i32 %a3, %b3
+
+ %i1 = add nuw nsw i64 %i, 1
+ %i2 = add nuw nsw i64 %i, 2
+ %i3 = add nuw nsw i64 %i, 3
+ %q0 = getelementptr inbounds i32, ptr %A, i64 %i
+ %q1 = getelementptr inbounds i32, ptr %A, i64 %i1
+ %q2 = getelementptr inbounds i32, ptr %A, i64 %i2
+ %q3 = getelementptr inbounds i32, ptr %A, i64 %i3
+ store i32 %v0, ptr %q0, align 4
+ store i32 %v1, ptr %q1, align 4
+ store i32 %v2, ptr %q2, align 4
+ store i32 %v3, ptr %q3, align 4
+
+ %next = add nuw nsw i64 %i, 4
+ %done = icmp uge i64 %next, %n
+ br i1 %done, label %exit, label %loop
+
+exit:
+ ret void
+}
>From de7e7ec09a14295aed1d379c2ccdb6e11b00d34b Mon Sep 17 00:00:00 2001
From: mbhade <mbhade at amd.com>
Date: Mon, 28 Sep 2026 14:36:45 +0530
Subject: [PATCH 21/21] [SLP] Correct STLF hazard charge accounting
Charge each distinct conflicting load and only retain a charge after its
vector load is emitted. Keying the committed charges by pointer also keeps
deduplication valid after vectorization erases the scalar load.
Co-authored-by: Cursor <cursoragent at cursor.com>
---
.../Transforms/Vectorize/SLPVectorizer.cpp | 115 +++++++++++-------
.../X86/memory-runtime-checks.ll | 12 +-
.../store-load-forward-multichain-dedup.ll | 61 +++++++++-
.../X86/store-load-forward-non-power-of-2.ll | 11 +-
.../X86/store-load-forward-two-loads-dedup.ll | 16 ++-
5 files changed, 152 insertions(+), 63 deletions(-)
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 9f4a9f39bbae5f..d7a57d775709c2 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -3821,28 +3821,32 @@ class slpvectorizer::BoUpSLP {
std::tuple<SmallVector<int>, VectorType *, unsigned, bool>>
CompressEntryToData;
- /// Loads that a store-to-load forwarding conflict penalty has already been
- /// charged for, across the whole function (the lifetime of this BoUpSLP
- /// object), because some earlier tree that conflicted with them was
- /// actually committed to vectorized IR. Several store or load tree entries
- /// -- in the same tree or in different trees/store chains -- can conflict
- /// with the same loop-carried load; hardware stalls once per load, not
- /// once per conflicting entry, so only the first *committed* tree pays.
- /// Deliberately never cleared during this object's lifetime (unlike the
- /// other buildTree-scoped state in deleteTree()): a dedup set that resets
- /// per attempt cannot catch the common case, since every store chain is
- /// costed as its own separate tree (see buildTreeRec()'s single-operand
- /// recursion), so there is never more than one charge to dedup within a
- /// single cost pass.
- SmallPtrSet<const LoadInst *, 8> STLFChargedLoads;
-
- /// Loads that the tree currently being costed would charge a store-to-load
- /// forwarding penalty for, staged here until the tree is actually
- /// vectorized (see vectorizeTree()), at which point they are promoted into
- /// STLFChargedLoads. Nothing is promoted -- and no later chain is ever
+ /// Pointer operands of loads that a store-to-load forwarding conflict
+ /// penalty has already been charged for, across the whole function (the
+ /// lifetime of this BoUpSLP object), because some earlier tree that
+ /// conflicted with them was actually committed to vectorized IR. Keyed by
+ /// the pointer operand rather than the LoadInst: vectorizeTree erases the
+ /// scalar loads, so a later tree sees a new load of the same address.
+ /// Several store or load tree entries -- in the same tree or in different
+ /// trees/store chains -- can conflict with that address; hardware stalls
+ /// once per load, not once per conflicting entry, so only the first
+ /// *committed* tree pays. Deliberately never cleared during this object's
+ /// lifetime (unlike the other buildTree-scoped state in deleteTree()): a
+ /// dedup set that resets per attempt cannot catch the common case, since
+ /// every store chain is costed as its own separate tree (see
+ /// buildTreeRec()'s single-operand recursion), so there is never more than
+ /// one charge to dedup within a single cost pass.
+ SmallPtrSet<const Value *, 8> STLFChargedLoads;
+
+ /// (Tree entry, load) pairs the tree currently being costed would charge a
+ /// store-to-load forwarding penalty for, staged here until vectorizeTree().
+ /// Only entries that are still emitted (not deleted or turned into a
+ /// gather while trimming) are promoted, and the recorded key is the load's
+ /// pointer operand. Nothing is promoted -- and no later chain is ever
/// under-charged -- if this tree is costed but rejected. Cleared per
/// buildTree attempt in deleteTree().
- SmallPtrSet<const LoadInst *, 8> PendingSTLFChargedLoads;
+ SmallVector<std::pair<const TreeEntry *, const LoadInst *>>
+ PendingSTLFChargedLoads;
/// The loop nest, used to check if only a single loop nest is vectorized, not
/// multiple, to avoid side-effects from the loop-aware cost model.
@@ -18100,14 +18104,14 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
E->State == TreeEntry::CompressVectorize) &&
(CostKind == TTI::TCK_RecipThroughput ||
CostKind == TTI::TCK_Latency)) {
- if (STLFChargedLoads.contains(STLFBaseLoad)) {
+ if (STLFChargedLoads.contains(STLFBaseLoad->getPointerOperand())) {
++NumSTLFChargesDeduped;
} else if (findStoreLoadForwardingHazardForLoad(STLFBaseLoad,
- HazardCheckVF)) {
+ HazardCheckVF)) {
Type *STLFVecTy = getWidenedType(LI0->getType(), STLFLoadVF);
VecLdCost +=
TTI->getStoreLoadForwardingConflictCost(STLFVecTy, CostKind);
- PendingSTLFChargedLoads.insert(STLFBaseLoad);
+ PendingSTLFChargedLoads.emplace_back(E, STLFBaseLoad);
}
}
return VecLdCost + CommonCost;
@@ -18193,8 +18197,9 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
// hazard, so only account for it under those cost kinds. Collect every
// conflicting load (not just the first) so a load already paid for by
// an earlier committed tree does not, on its own, cause a second
- // charge here: only a load this store is the *first* committed
- // conflict for should add the penalty and be staged for promotion.
+ // charge here: each load this store is the *first* committed conflict
+ // for adds its own penalty and is staged for promotion. A load an
+ // earlier committed tree already paid for does not.
//
// Covers Vectorize (contiguous window) and, when the intra-vector lane
// stride resolves to a compile-time constant, StridedVectorize (real
@@ -18236,20 +18241,29 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
(CostKind == TTI::TCK_RecipThroughput ||
CostKind == TTI::TCK_Latency)) {
SmallVector<LoadInst *> ConflictingLoads;
+ // Contiguous Vectorize stores are emitted at VL0. Reverse
+ // StridedVectorize rebinds the pointer to ReorderIndices.front()
+ // (BaseSI); other states keep VL0.
+ StoreInst *STLFBaseStore = cast<StoreInst>(VL0);
+ if (E->State == TreeEntry::StridedVectorize && IsReorder &&
+ isReverseOrder(E->ReorderIndices))
+ STLFBaseStore = BaseSI;
if (findStoreLoadForwardingConflict(
- BaseSI, StoreSTLFVF,
+ STLFBaseStore, StoreSTLFVF,
/*OnlyLoad=*/nullptr, /*LoadSizeOverride=*/std::nullopt,
&ConflictingLoads, StoreSizeOverride)) {
- bool AnyUncharged = false;
- for (LoadInst *LI : ConflictingLoads)
- AnyUncharged |= !STLFChargedLoads.contains(LI);
- if (AnyUncharged) {
- VecStCost += TTI->getStoreLoadForwardingConflictCost(VecTy, CostKind);
- for (LoadInst *LI : ConflictingLoads)
- PendingSTLFChargedLoads.insert(LI);
- } else {
- ++NumSTLFChargesDeduped;
+ unsigned NewConflicts = 0;
+ for (LoadInst *LI : ConflictingLoads) {
+ if (STLFChargedLoads.contains(LI->getPointerOperand()))
+ continue;
+ ++NewConflicts;
+ PendingSTLFChargedLoads.emplace_back(E, LI);
}
+ if (NewConflicts)
+ VecStCost += NewConflicts * TTI->getStoreLoadForwardingConflictCost(
+ VecTy, CostKind);
+ else
+ ++NumSTLFChargesDeduped;
}
}
return VecStCost + CommonCost;
@@ -25717,8 +25731,13 @@ Value *BoUpSLP::vectorizeTree(
// (store chains, lists, reductions) funnels through this one function, so
// this is the single chokepoint to hook. A tree that was only costed and
// then rejected never reaches here, so it never suppresses a later
- // chain's real charge.
- STLFChargedLoads.insert_range(PendingSTLFChargedLoads);
+ // chain's real charge. Entries trimmed away or turned into gathers did
+ // not emit the wide access the penalty was priced for, so they must not
+ // suppress a later chain. Record the pointer operand: the LoadInst itself
+ // is erased by the emission below.
+ for (auto [TE, Ld] : PendingSTLFChargedLoads)
+ if (!DeletedNodes.contains(TE) && !TransformedToGatherNodes.contains(TE))
+ STLFChargedLoads.insert(Ld->getPointerOperand());
PendingSTLFChargedLoads.clear();
// Clean Entry-to-LastInstruction table. It can be affected after scheduling,
// need to rebuild it.
@@ -29124,7 +29143,12 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
std::optional<uint64_t> StoreSizeOverride) {
assert(BaseStore && "Expected a valid base store");
- Type *ValueTy = BaseStore->getValueOperand()->getType();
+ Type *StoreTy = getValueType(BaseStore);
+ // A scalable stored type has no fixed byte window. Bail out before
+ // getNumElements, which rejects scalable vectors.
+ if (isa<ScalableVectorType>(StoreTy))
+ return false;
+ Type *ValueTy = StoreTy->getScalarType();
TypeSize StoreSize = DL->getTypeStoreSize(ValueTy);
if (StoreSize.isScalable())
return false;
@@ -29137,8 +29161,9 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
if (!StoreL)
return false;
- uint64_t VectorStoreBytes =
- StoreSizeOverride ? *StoreSizeOverride : VF * ElementSize;
+ uint64_t VectorStoreBytes = StoreSizeOverride
+ ? *StoreSizeOverride
+ : VF * getNumElements(StoreTy) * ElementSize;
LLVM_DEBUG(dbgs() << "SLP: STLF check: VF=" << VF
<< " ElementSize=" << ElementSize
<< " VectorStoreBytes=" << VectorStoreBytes << "\n");
@@ -29158,11 +29183,11 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
// modeled at scalar width.
Value *StoreBase = getUnderlyingObject(BaseStore->getPointerOperand());
const auto CandidateLoads = [&] {
- SmallPtrSet<LoadInst *, 8> Loads;
+ SmallVector<LoadInst *> Loads;
if (OnlyLoad) {
if (OnlyLoad->isSimple() &&
getUnderlyingObject(OnlyLoad->getPointerOperand()) == StoreBase)
- Loads.insert(OnlyLoad);
+ Loads.push_back(OnlyLoad);
return Loads;
}
for (BasicBlock *BB : StoreL->blocks())
@@ -29170,7 +29195,7 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
if (auto *LoadI = dyn_cast<LoadInst>(&I))
if (LoadI->isSimple() &&
getUnderlyingObject(LoadI->getPointerOperand()) == StoreBase)
- Loads.insert(LoadI);
+ Loads.push_back(LoadI);
return Loads;
}();
@@ -29283,8 +29308,8 @@ bool BoUpSLP::findStoreLoadForwardingConflict(
// conservative and do not treat it as a current-iteration hazard.
bool StoreBeforeLoad = BaseStore->getParent() == LoadI->getParent() &&
BaseStore->comesBefore(LoadI);
- bool OverlapsCurrentStore =
- IsWidenedBaseLane && Distance < LoadElementSize && StoreBeforeLoad;
+ bool OverlapsCurrentStore = (!WidenedLoadEntry || IsWidenedBaseLane) &&
+ Distance < LoadElementSize && StoreBeforeLoad;
if (!OverlapsCurrentStore && StoreStride && LoadStride &&
*StoreStride == *LoadStride && *StoreStride > 0) {
int64_t Stride = *StoreStride;
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
index 8712fa4beec9c8..b0b27975216bef 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/memory-runtime-checks.ll
@@ -210,24 +210,28 @@ define void @gather_sequence_crash(<2 x float> %arg, ptr %arg1, float %arg2, ptr
; CHECK-NEXT: [[TMP24:%.*]] = getelementptr float, ptr [[ARG4:%.*]], i64 7
; CHECK-NEXT: br i1 [[C_2:%.*]], label [[BB25:%.*]], label [[BB22]]
; CHECK: bb25:
+; CHECK-NEXT: [[TMP26:%.*]] = getelementptr float, ptr [[ARG4]], i64 6
; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP24]], align 4
; CHECK-NEXT: [[TMP27:%.*]] = load float, ptr [[ARG5:%.*]], align 4
-; CHECK-NEXT: [[TMP28:%.*]] = getelementptr float, ptr [[ARG4]], i64 5
; CHECK-NEXT: [[TMP29:%.*]] = fadd float 0.000000e+00, 0.000000e+00
+; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP26]], align 4
; CHECK-NEXT: [[TMP30:%.*]] = getelementptr float, ptr [[ARG4]], i64 4
-; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP28]], align 4
; CHECK-NEXT: [[TMP31:%.*]] = fadd float 0.000000e+00, 0.000000e+00
-; CHECK-NEXT: store float 0.000000e+00, ptr [[TMP30]], align 4
+; CHECK-NEXT: store <2 x float> zeroinitializer, ptr [[TMP30]], align 4
; CHECK-NEXT: br label [[BB33:%.*]]
; CHECK: bb33:
; CHECK-NEXT: br label [[BB34:%.*]]
; CHECK: bb34:
; CHECK-NEXT: [[TMP35:%.*]] = getelementptr float, ptr [[ARG4]], i64 3
+; CHECK-NEXT: [[TMP36:%.*]] = getelementptr float, ptr [[ARG4]], i64 2
; CHECK-NEXT: [[TMP37:%.*]] = load float, ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP38:%.*]] = fadd float 0.000000e+00, [[TMP37]]
; CHECK-NEXT: store float [[TMP38]], ptr [[TMP35]], align 4
; CHECK-NEXT: [[TMP39:%.*]] = getelementptr float, ptr [[ARG4]], i64 1
-; CHECK-NEXT: [[TMP5:%.*]] = load <2 x float>, ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP40:%.*]] = load float, ptr [[TMP36]], align 4
+; CHECK-NEXT: [[TMP42:%.*]] = load float, ptr [[TMP39]], align 4
+; CHECK-NEXT: [[TMP9:%.*]] = insertelement <2 x float> poison, float [[TMP42]], i64 0
+; CHECK-NEXT: [[TMP5:%.*]] = insertelement <2 x float> [[TMP9]], float [[TMP40]], i64 1
; CHECK-NEXT: [[TMP7:%.*]] = fadd <2 x float> zeroinitializer, [[TMP5]]
; CHECK-NEXT: store <2 x float> [[TMP7]], ptr [[TMP39]], align 4
; CHECK-NEXT: [[TMP44:%.*]] = load float, ptr [[ARG3:%.*]], align 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll
index 4b81d424a7b440..12f836d07cf0ca 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-multichain-dedup.ll
@@ -1,12 +1,26 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
; RUN: opt < %s -passes=slp-vectorizer -S -slp-max-reg-size=128 \
-; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=13 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=5 \
; RUN: | FileCheck %s
;
+; The same function at a penalty high enough to trim the load subtree. The
+; trimmed entry must not be recorded as paid, so the load is not widened on a
+; later attempt and the waiver statistic stays silent. The stores still
+; vectorize.
+; RUN: opt < %s -passes=slp-vectorizer -S -slp-max-reg-size=128 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=13 \
+; RUN: | FileCheck %s --check-prefix=TRIMMED
+;
; REQUIRES: asserts
; RUN: opt < %s -passes=slp-vectorizer -disable-output -slp-max-reg-size=128 \
-; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=13 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=5 \
; RUN: -stats 2>&1 | FileCheck %s --check-prefix=STATS
+; RUN: opt < %s -passes=slp-vectorizer -disable-output -slp-max-reg-size=128 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=13 \
+; RUN: -stats 2>&1 | FileCheck %s --check-prefix=TRIM-STATS
+;
+; TRIM-STATS-NOT: waived because an earlier committed tree already paid for the
+; TRIM-STATS-NOT: same load
;
; STATS: {{[0-9]+}} SLP - Number of store-to-load forwarding hazard charges
; STATS-SAME: waived because an earlier committed tree already paid for the
@@ -26,8 +40,11 @@ target triple = "x86_64-unknown-linux-gnu"
; store-to-load forwarding check on each of them.
;
; The four backward loads A[i-5..i-2] feed a chain of ops (xor/mul/add/xor)
-; profitable enough to widen into one <4 x i32> load despite also paying its
-; own STLF hazard-for-load charge. Once widened, the load is 16 bytes wide and
+; that still widens into one <4 x i32> load at this penalty, and that entry
+; is emitted (not trimmed), so its charge is promoted. The TRIMMED run uses
+; penalty 13, where that load entry is trimmed before vectorizeTree and must
+; not be recorded as paid; the <4 x i32> load stays unformed. Once widened, the
+; load is 16 bytes wide and
; straddles the boundary between this iteration's two 16-byte store windows in
; a future iteration (distance 20 bytes from chain A's base, 36 bytes from
; chain B's base; both misaligned to the 16-byte window).
@@ -65,6 +82,42 @@ define void @stlf_multichain_dedup(ptr noalias %A, ptr noalias %Out, i64 %n) {
; CHECK: [[EXIT]]:
; CHECK-NEXT: ret void
;
+; TRIMMED-LABEL: define void @stlf_multichain_dedup(
+; TRIMMED-SAME: ptr noalias [[A:%.*]], ptr noalias [[OUT:%.*]], i64 [[N:%.*]]) #[[ATTR0:[0-9]+]] {
+; TRIMMED-NEXT: [[ENTRY:.*]]:
+; TRIMMED-NEXT: br label %[[LOOP:.*]]
+; TRIMMED: [[LOOP]]:
+; TRIMMED-NEXT: [[I:%.*]] = phi i64 [ 8, %[[ENTRY]] ], [ [[NEXT:%.*]], %[[LOOP]] ]
+; TRIMMED-NEXT: [[LM5:%.*]] = add i64 [[I]], -5
+; TRIMMED-NEXT: [[LM4:%.*]] = add i64 [[I]], -4
+; TRIMMED-NEXT: [[LM3:%.*]] = add i64 [[I]], -3
+; TRIMMED-NEXT: [[LP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[LM5]]
+; TRIMMED-NEXT: [[LP1:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[LM4]]
+; TRIMMED-NEXT: [[LP2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[LM3]]
+; TRIMMED-NEXT: [[O0:%.*]] = getelementptr inbounds i32, ptr [[OUT]], i64 [[I]]
+; TRIMMED-NEXT: [[TMP0:%.*]] = load <2 x i32>, ptr [[LP2]], align 4
+; TRIMMED-NEXT: [[L1:%.*]] = load i32, ptr [[LP1]], align 4
+; TRIMMED-NEXT: [[L0:%.*]] = load i32, ptr [[LP0]], align 4
+; TRIMMED-NEXT: [[TMP1:%.*]] = insertelement <4 x i32> poison, i32 [[L0]], i64 0
+; TRIMMED-NEXT: [[TMP2:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[L1]], i64 1
+; TRIMMED-NEXT: [[TMP3:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
+; TRIMMED-NEXT: [[TMP4:%.*]] = shufflevector <4 x i32> [[TMP2]], <4 x i32> [[TMP3]], <4 x i32> <i32 0, i32 1, i32 4, i32 5>
+; TRIMMED-NEXT: [[TMP5:%.*]] = xor <4 x i32> [[TMP4]], <i32 17, i32 19, i32 23, i32 29>
+; TRIMMED-NEXT: [[TMP6:%.*]] = mul <4 x i32> [[TMP5]], <i32 3, i32 5, i32 7, i32 9>
+; TRIMMED-NEXT: [[TMP7:%.*]] = add <4 x i32> [[TMP6]], <i32 31, i32 37, i32 41, i32 43>
+; TRIMMED-NEXT: [[TMP8:%.*]] = xor <4 x i32> [[TMP7]], <i32 47, i32 53, i32 59, i32 61>
+; TRIMMED-NEXT: store <4 x i32> [[TMP8]], ptr [[O0]], align 4
+; TRIMMED-NEXT: [[A0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
+; TRIMMED-NEXT: store <4 x i32> <i32 111, i32 222, i32 333, i32 444>, ptr [[A0]], align 4
+; TRIMMED-NEXT: [[I4:%.*]] = add nuw nsw i64 [[I]], 4
+; TRIMMED-NEXT: [[B0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
+; TRIMMED-NEXT: store <4 x i32> <i32 555, i32 666, i32 777, i32 888>, ptr [[B0]], align 4
+; TRIMMED-NEXT: [[NEXT]] = add nuw nsw i64 [[I]], 8
+; TRIMMED-NEXT: [[DONE:%.*]] = icmp uge i64 [[NEXT]], [[N]]
+; TRIMMED-NEXT: br i1 [[DONE]], label %[[EXIT:.*]], label %[[LOOP]]
+; TRIMMED: [[EXIT]]:
+; TRIMMED-NEXT: ret void
+;
entry:
br label %loop
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
index 1a47422065d914..9dc667658c0a4d 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-non-power-of-2.ll
@@ -39,9 +39,13 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON: [[FOR_BODY]]:
; STLF-ON-NEXT: [[I:%.*]] = phi i64 [ 12, %[[ENTRY]] ], [ [[I_NEXT:%.*]], %[[FOR_BODY]] ]
; STLF-ON-NEXT: [[B0:%.*]] = add i64 [[I]], -12
+; STLF-ON-NEXT: [[B2:%.*]] = add i64 [[I]], -10
+; STLF-ON-NEXT: [[B3:%.*]] = add i64 [[I]], -9
; STLF-ON-NEXT: [[B4:%.*]] = add i64 [[I]], -8
; STLF-ON-NEXT: [[B5:%.*]] = add i64 [[I]], -7
; STLF-ON-NEXT: [[P0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B0]]
+; STLF-ON-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B2]]
+; STLF-ON-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B3]]
; STLF-ON-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B4]]
; STLF-ON-NEXT: [[P5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[B5]]
; STLF-ON-NEXT: [[L6:%.*]] = load i32, ptr [[P6]], align 4
@@ -51,7 +55,12 @@ define void @stlf_non_power_of_2(ptr noalias %A, i64 %n) {
; STLF-ON-NEXT: [[GEP0:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I]]
; STLF-ON-NEXT: [[GEP6:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I4]]
; STLF-ON-NEXT: [[GEP5:%.*]] = getelementptr inbounds i32, ptr [[A]], i64 [[I5]]
-; STLF-ON-NEXT: [[TMP3:%.*]] = load <4 x i32>, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[L3:%.*]] = load i32, ptr [[P3]], align 4
+; STLF-ON-NEXT: [[L2:%.*]] = load i32, ptr [[P2]], align 4
+; STLF-ON-NEXT: [[TMP0:%.*]] = load <2 x i32>, ptr [[P0]], align 4
+; STLF-ON-NEXT: [[TMP1:%.*]] = shufflevector <2 x i32> [[TMP0]], <2 x i32> poison, <4 x i32> <i32 0, i32 1, i32 poison, i32 poison>
+; STLF-ON-NEXT: [[TMP2:%.*]] = insertelement <4 x i32> [[TMP1]], i32 [[L2]], i64 2
+; STLF-ON-NEXT: [[TMP3:%.*]] = insertelement <4 x i32> [[TMP2]], i32 [[L3]], i64 3
; STLF-ON-NEXT: [[TMP4:%.*]] = add nsw <4 x i32> [[TMP3]], <i32 1, i32 2, i32 3, i32 4>
; STLF-ON-NEXT: store <4 x i32> [[TMP4]], ptr [[GEP0]], align 4
; STLF-ON-NEXT: store i32 [[T7]], ptr [[GEP6]], align 4
diff --git a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll
index 0a4fa03feb2c56..fd403ff9fd7684 100644
--- a/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll
+++ b/llvm/test/Transforms/SLPVectorizer/X86/store-load-forward-two-loads-dedup.ll
@@ -1,10 +1,10 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
; REQUIRES: asserts
; RUN: opt < %s -passes=slp-vectorizer -S \
-; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=12 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=11 \
; RUN: | FileCheck %s
; RUN: opt < %s -passes=slp-vectorizer -disable-output \
-; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=12 \
+; RUN: -mtriple=x86_64-- -mcpu=znver5 -store-load-forwarding-conflict-cost=11 \
; RUN: -stats 2>&1 | FileCheck %s --check-prefix=STATS
;
@@ -17,13 +17,11 @@
; conflicting load, not just the first) against silently dropping a second,
; distinct hazard.
;
-; At this exact penalty (12), two independent charges (24) outweigh the
-; natural vectorization savings and the store correctly stays scalar; if the
-; two distinct loads were wrongly deduped down to a single charge (12), the
-; tree would incorrectly still look profitable and widen (see the sibling
-; store-load-forward-multichain-dedup.ll test for the "only pay once"
-; direction of this same dedup logic, where the shared load legitimately
-; collapses to one charge).
+; At this penalty (11), one charge still vectorizes (Found cost -2 on znver5)
+; but a charge per conflicting load does not, so the store stays scalar. A
+; single shared charge for both loads would widen. The sibling
+; store-load-forward-multichain-dedup.ll test covers the other direction:
+; one load conflicting with two store chains is charged once.
;
; STATS-NOT: {{[0-9]+}} SLP - Number of store-to-load forwarding hazard charges
;
More information about the llvm-commits
mailing list