[llvm] 62db1c8 - [SLP]Better decision making on whether to try stores packs for vectorization

[Alexey Bataev via llvm-commits]

Author: Alexey Bataev
Date: 2024-11-07T14:23:15-08:00
New Revision: 62db1c8a076c7167e404412182f4a8915f4ff6ee

URL: https://github.com/llvm/llvm-project/commit/62db1c8a076c7167e404412182f4a8915f4ff6ee
DIFF: https://github.com/llvm/llvm-project/commit/62db1c8a076c7167e404412182f4a8915f4ff6ee.diff

LOG: [SLP]Better decision making on whether to try stores packs for vectorization

Since the stores are sorted by distance, comparing the indices in the
original array and early exit, if the index is less than the index of
the last store, not always the best strategy. Better to remove such
stores explicitly to try better to check for the vectorization
opportunity.

Fixes #115008

Added: 
    

Modified: 
    llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
    llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll

Removed: 
    


################################################################################
diff  --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index be7ddeb89e789f..b2f677fb84f983 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -18515,25 +18515,30 @@ bool SLPVectorizerPass::vectorizeStores(
       }
       // Try to vectorize the first found set to avoid duplicate analysis.
       TryToVectorize(Set.second);
+      unsigned ItIdx = It->first;
+      int ItDist = It->second;
       StoreIndexToDistSet PrevSet;
-      PrevSet.swap(Set.second);
+      copy_if(Set.second, std::inserter(PrevSet, PrevSet.end()),
+              [&](const std::pair<unsigned, int> &Pair) {
+                return Pair.first > ItIdx;
+              });
+      Set.second.clear();
       Set.first = Idx;
       Set.second.emplace(Idx, 0);
       // Insert stores that followed previous match to try to vectorize them
       // with this store.
-      unsigned StartIdx = It->first + 1;
+      unsigned StartIdx = ItIdx + 1;
       SmallBitVector UsedStores(Idx - StartIdx);
       // Distances to previously found dup store (or this store, since they
       // store to the same addresses).
       SmallVector<int> Dists(Idx - StartIdx, 0);
       for (const std::pair<unsigned, int> &Pair : reverse(PrevSet)) {
         // Do not try to vectorize sequences, we already tried.
-        if (Pair.first <= It->first ||
-            VectorizedStores.contains(Stores[Pair.first]))
+        if (VectorizedStores.contains(Stores[Pair.first]))
           break;
         unsigned BI = Pair.first - StartIdx;
         UsedStores.set(BI);
-        Dists[BI] = Pair.second - It->second;
+        Dists[BI] = Pair.second - ItDist;
       }
       for (unsigned I = StartIdx; I < Idx; ++I) {
         unsigned BI = I - StartIdx;

diff  --git a/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll b/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll
index f126192271cd95..48928d2dfd4738 100644
--- a/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll
+++ b/llvm/test/Transforms/SLPVectorizer/RISCV/repeated-address-store.ll
@@ -6,12 +6,7 @@ define void @test(ptr %dest) {
 ; CHECK-SAME: ptr [[DEST:%.*]]) #[[ATTR0:[0-9]+]] {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[INC3:%.*]] = getelementptr inbounds i32, ptr [[DEST]], i64 3
-; CHECK-NEXT:    store i32 1, ptr [[INC3]], align 2
-; CHECK-NEXT:    store i32 1, ptr [[DEST]], align 4
-; CHECK-NEXT:    [[INC1:%.*]] = getelementptr inbounds i32, ptr [[DEST]], i64 1
-; CHECK-NEXT:    store i32 1, ptr [[INC1]], align 2
-; CHECK-NEXT:    [[INC2:%.*]] = getelementptr inbounds i32, ptr [[DEST]], i64 2
-; CHECK-NEXT:    store i32 1, ptr [[INC2]], align 2
+; CHECK-NEXT:    store <4 x i32> splat (i32 1), ptr [[DEST]], align 4
 ; CHECK-NEXT:    store i32 2, ptr [[DEST]], align 2
 ; CHECK-NEXT:    store i32 1, ptr [[INC3]], align 2
 ; CHECK-NEXT:    ret void