[llvm] 62db1c8 - [SLP]Better decision making on whether to try stores packs for vectorization
Alexey Bataev via llvm-commits
llvm-commits at lists.llvm.org
Thu Nov 7 14:23:25 PST 2024
Author: Alexey Bataev
Date: 2024-11-07T14:23:15-08:00
New Revision: 62db1c8a076c7167e404412182f4a8915f4ff6ee
URL: https://github.com/llvm/llvm-project/commit/62db1c8a076c7167e404412182f4a8915f4ff6ee
DIFF: https://github.com/llvm/llvm-project/commit/62db1c8a076c7167e404412182f4a8915f4ff6ee.diff
LOG: [SLP]Better decision making on whether to try stores packs for vectorization
Since the stores are sorted by distance, comparing the indices in the
original array and early exit, if the index is less than the index of
the last store, not always the best strategy. Better to remove such
stores explicitly to try better to check for the vectorization
opportunity.
Fixes #115008
Added:
Modified:
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll
Removed:
################################################################################
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index be7ddeb89e789f..b2f677fb84f983 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -18515,25 +18515,30 @@ bool SLPVectorizerPass::vectorizeStores(
}
// Try to vectorize the first found set to avoid duplicate analysis.
TryToVectorize(Set.second);
+ unsigned ItIdx = It->first;
+ int ItDist = It->second;
StoreIndexToDistSet PrevSet;
- PrevSet.swap(Set.second);
+ copy_if(Set.second, std::inserter(PrevSet, PrevSet.end()),
+ [&](const std::pair<unsigned, int> &Pair) {
+ return Pair.first > ItIdx;
+ });
+ Set.second.clear();
Set.first = Idx;
Set.second.emplace(Idx, 0);
// Insert stores that followed previous match to try to vectorize them
// with this store.
- unsigned StartIdx = It->first + 1;
+ unsigned StartIdx = ItIdx + 1;
SmallBitVector UsedStores(Idx - StartIdx);
// Distances to previously found dup store (or this store, since they
// store to the same addresses).
SmallVector<int> Dists(Idx - StartIdx, 0);
for (const std::pair<unsigned, int> &Pair : reverse(PrevSet)) {
// Do not try to vectorize sequences, we already tried.
- if (Pair.first <= It->first ||
- VectorizedStores.contains(Stores[Pair.first]))
+ if (VectorizedStores.contains(Stores[Pair.first]))
break;
unsigned BI = Pair.first - StartIdx;
UsedStores.set(BI);
- Dists[BI] = Pair.second - It->second;
+ Dists[BI] = Pair.second - ItDist;
}
for (unsigned I = StartIdx; I < Idx; ++I) {
unsigned BI = I - StartIdx;
diff --git a/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll b/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll
index f126192271cd95..48928d2dfd4738 100644
--- a/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll
+++ b/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll
@@ -6,12 +6,7 @@ define void @test(ptr %dest) {
; CHECK-SAME: ptr [[DEST:%.*]]) #[[ATTR0:[0-9]+]] {
; CHECK-NEXT: [[ENTRY:.*:]]
; CHECK-NEXT: [[INC3:%.*]] = getelementptr inbounds i32, ptr [[DEST]], i64 3
-; CHECK-NEXT: store i32 1, ptr [[INC3]], align 2
-; CHECK-NEXT: store i32 1, ptr [[DEST]], align 4
-; CHECK-NEXT: [[INC1:%.*]] = getelementptr inbounds i32, ptr [[DEST]], i64 1
-; CHECK-NEXT: store i32 1, ptr [[INC1]], align 2
-; CHECK-NEXT: [[INC2:%.*]] = getelementptr inbounds i32, ptr [[DEST]], i64 2
-; CHECK-NEXT: store i32 1, ptr [[INC2]], align 2
+; CHECK-NEXT: store <4 x i32> splat (i32 1), ptr [[DEST]], align 4
; CHECK-NEXT: store i32 2, ptr [[DEST]], align 2
; CHECK-NEXT: store i32 1, ptr [[INC3]], align 2
; CHECK-NEXT: ret void
More information about the llvm-commits
mailing list