[llvm-branch-commits] [llvm] [SLP] Price gathered lanes of vectorized entries as a shuffle (PR #240136)
Alexey Bataev via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Sun Nov 8 09:35:36 PST 2026
================
@@ -21932,6 +21932,26 @@ BoUpSLP::isGatherShuffledSingleRegisterEntry(
getScalarizationOverhead(*TTI, Opts.slp_revec, VL.front()->getType(),
MaskVecTy, DemandedElts, /*Insert=*/true,
/*Extract=*/false, CostKind);
+ // A lane taken from another vectorized entry needs a shuffle, which a zero
+ // insert price misses.
+ APInt VectorizedElts = APInt::getZero(SubMask.size());
+ SmallVector<int> VectorizedMask(SubMask.size(), PoisonMaskElem);
+ for (auto [I, Idx] : enumerate(SubMask)) {
+ if (Idx == PoisonMaskElem || !isVectorized(VL[I]))
+ continue;
+ VectorizedElts.setBit(I);
+ VectorizedMask[I] = Idx;
+ }
+ if (!VectorizedElts.isZero() &&
+ getScalarizationOverhead(*TTI, Opts.slp_revec, VL.front()->getType(),
+ MaskVecTy, VectorizedElts, /*Insert=*/true,
+ /*Extract=*/false,
+ CostKind) == TTI::TCC_Free) {
----------------
alexey-bataev wrote:
The == TTI::TCC_Free check covers all vectorized lanes together. One vectorized lane at a position where insert is not free turns off the correction for every lane, although every lane still needs its extract.
https://github.com/llvm/llvm-project/pull/240136
More information about the llvm-branch-commits
mailing list