[llvm] [SLP] Split blocking build-vector stores in scalar chains (PR #194970)

[Yaxun Liu via llvm-commits]

================
@@ -27346,6 +27473,137 @@ bool SLPVectorizerPass::vectorizeStores(
                         return A && (!B || A->getStride() < B->getStride());
                       });
 
+    // Mixed scalar/build-vector slice handling.
+    //
+    // Conceptually a "store reduction": SLP builds a tree on the per-lane
+    // values, treating the original IE chain and StoreInsts together as
+    // the reduction-operation chain via UserIgnoreList. After the tree's
+    // standard cost analysis we add the store-sink-specific cost (one vector
+    // store, minus the replaced scalar store costs); if profitable we let
+    // the standard tree emitter materialize the lane-value vector, then
+    // emit the new <VF x T> store and erase the originals here. The IE
+    // chains feeding any erased build-vector store become trivially dead
+    // and are cleaned up by the standard pass at the end.
+    auto VectorizeBuildVectorStoreSinkSlice =
+        [&](ArrayRef<StoreLane> Chain) -> std::optional<bool> {
+      SmallVector<Value *> LaneValues;
+      SmallVector<std::pair<StoreInst *, unsigned>> Owners;
+      StoreInst *LastOwner = nullptr;
+      for (const StoreLane &Lane : Chain) {
+        LaneValues.push_back(Lane.ScalarValue);
+        if (Lane.Store == LastOwner) {
+          Owners.back().second++;
+        } else {
+          Owners.emplace_back(Lane.Store, 1);
+          LastOwner = Lane.Store;
+        }
+      }
+      // Each build-vector owner must be fully covered by this slice;
+      // otherwise erasing it would lose data outside the slice.
+      for (auto &[SI, Count] : Owners) {
+        SmallVector<Value *, 16> Elts;
+        SmallVector<Instruction *, 16> Insts;
+        if (collectBuildVector(SI->getValueOperand(), Elts, Insts) &&
+            Count != Elts.size())
+          return false;
+      }
+      // UserIgnoreList: original stores plus any IE chain feeding a build-
+      // vector store. SLP treats this exactly like a reduction tree: the
+      // lane values' uses inside listed instructions are not counted as
+      // external uses, since those instructions will be erased after the
+      // new vector store is emitted.
+      SmallDenseSet<Value *> Ignored;
+      for (auto &[SI, _] : Owners) {
+        Ignored.insert(SI);
+        SmallVector<Value *, 16> Elts;
+        SmallVector<Instruction *, 16> Insts;
+        if (collectBuildVector(SI->getValueOperand(), Elts, Insts))
+          for (Instruction *I : Insts)
+            Ignored.insert(I);
+      }
+      // Conservatively require owner stores to appear in the same order in the
+      // block as they appear in the lane range, and reject any intervening
+      // instruction that may read or write memory. The standard scalar store
+      // path gets memory-dependence checks through store-root scheduling; this
+      // manual store sink must not move a wider store across unknown memory.
+      StoreInst *PrevStore = nullptr;
+      for (auto &[SI, _] : Owners) {
+        if (PrevStore && !PrevStore->comesBefore(SI))
+          return false;
+        PrevStore = SI;
+      }
+      StoreInst *FirstStore = Owners.front().first;
+      StoreInst *LastStore = Owners.back().first;
+      for (Instruction &I : make_range(std::next(FirstStore->getIterator()),
+                                       LastStore->getIterator())) {
+        if (Ignored.contains(&I))
+          continue;
+        if (I.mayReadOrWriteMemory())
+          return false;
+      }
+      R.buildTree(LaneValues, Ignored);
----------------
yxsamliu wrote:

Fixed, thanks for the report.

The issue was that this path uses `UserIgnoreList` for ignored stores/build-vector instructions, but the min-bitwidth code treated it like a reduction list and tried to query the size of a void-typed store. I added the test case and now skip that reduction-only analysis unless the ignored values are sized integers.

https://github.com/llvm/llvm-project/pull/194970