[llvm] [CostModel] Add a DstTy to getShuffleCost (PR #141634)
David Green via llvm-commits
llvm-commits at lists.llvm.org
Fri May 30 07:28:31 PDT 2025
https://github.com/davemgreen updated https://github.com/llvm/llvm-project/pull/141634
>From 82772883b84db2c79a682261f8e42ca6e6215120 Mon Sep 17 00:00:00 2001
From: David Green <david.green at arm.com>
Date: Tue, 13 May 2025 12:56:07 +0100
Subject: [PATCH 1/2] [CostModel] Add a DstTy to getShuffleCost
A shuffle will take two input vectors and a mask, to produce a new vector of
size <MaskElts x SrcEltTy>. Historically it has been assumed that the SrcTy and
the DstTy are the same for getShuffleCost, with that being relaxed in recent
years. If the Tp passed to getShuffleCost is the SrcTy, then the DstTy can be
calculated from the Mask elts and the src elt size, but the Mask is not always
provided and the Tp is not reliably always the SrcTy. This has led to
situations notably in the SLP vectorizer but also in the generic cost routines
where assumption about how vectors will be legalized are built into the generic
cost routines - for example whether they will widen or promote, with the cost
modelling assuming they will widen but the default lowering to promote for
integer vectors.
This patch attempts to start improving that - it originally tried to alter more
of the cost model but that too quickly became too many changes at once, so this
patch just plumbs in a DstTy to getShuffleCost so that DstTy and SrcTy can be
reliably distinguished. The callers of getShuffleCost have been updated to try
and include a DstTy that is more accurate. Otherwise it tries to be fairly
non-functional, keeping the SrcTy used as the primary type used in shuffle
cost routines, only using DstTy where it was in the past (for InsertSubVector
for example).
Some asserts have been added that help to check for consistent values when a
Mask and a DstTy are provided to getShuffleCost. Some of them took a while to
get right, and some non-mask calls might still be incorrect. Hopefully this
will provide a useful base to build more shuffles that alter size.
---
.../llvm/Analysis/TargetTransformInfo.h | 28 ++--
.../llvm/Analysis/TargetTransformInfoImpl.h | 60 ++++---
llvm/include/llvm/CodeGen/BasicTTIImpl.h | 60 +++----
llvm/lib/Analysis/TargetTransformInfo.cpp | 11 +-
.../AArch64/AArch64TargetTransformInfo.cpp | 58 ++++---
.../AArch64/AArch64TargetTransformInfo.h | 6 +-
.../AMDGPU/AMDGPUTargetTransformInfo.cpp | 17 +-
.../Target/AMDGPU/AMDGPUTargetTransformInfo.h | 6 +-
.../lib/Target/ARM/ARMTargetTransformInfo.cpp | 27 +--
llvm/lib/Target/ARM/ARMTargetTransformInfo.h | 6 +-
.../Hexagon/HexagonTargetTransformInfo.cpp | 10 +-
.../Hexagon/HexagonTargetTransformInfo.h | 6 +-
.../Target/PowerPC/PPCTargetTransformInfo.cpp | 7 +-
.../Target/PowerPC/PPCTargetTransformInfo.h | 6 +-
.../Target/RISCV/RISCVTargetTransformInfo.cpp | 70 ++++----
.../Target/RISCV/RISCVTargetTransformInfo.h | 6 +-
.../SystemZ/SystemZTargetTransformInfo.cpp | 19 ++-
.../SystemZ/SystemZTargetTransformInfo.h | 6 +-
.../lib/Target/X86/X86TargetTransformInfo.cpp | 151 +++++++++--------
llvm/lib/Target/X86/X86TargetTransformInfo.h | 6 +-
.../Scalar/LowerMatrixIntrinsics.cpp | 12 +-
.../Transforms/Vectorize/LoopVectorize.cpp | 13 +-
.../Transforms/Vectorize/SLPVectorizer.cpp | 23 ++-
.../lib/Transforms/Vectorize/VPlanRecipes.cpp | 23 +--
.../Transforms/Vectorize/VectorCombine.cpp | 155 +++++++++---------
25 files changed, 444 insertions(+), 348 deletions(-)
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index 8f4ce80ada5ed..6335efc243813 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1381,19 +1381,21 @@ class TargetTransformInfo {
const SmallBitVector &OpcodeMask,
TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput) const;
- /// \return The cost of a shuffle instruction of kind Kind and of type Tp.
- /// The exact mask may be passed as Mask, or else the array will be empty.
- /// The index and subtype parameters are used by the subvector insertion and
- /// extraction shuffle kinds to show the insert/extract point and the type of
- /// the subvector being inserted/extracted. The operands of the shuffle can be
- /// passed through \p Args, which helps improve the cost estimation in some
- /// cases, like in broadcast loads.
- /// NOTE: For subvector extractions Tp represents the source type.
- LLVM_ABI InstructionCost getShuffleCost(
- ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask = {},
- TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput, int Index = 0,
- VectorType *SubTp = nullptr, ArrayRef<const Value *> Args = {},
- const Instruction *CxtI = nullptr) const;
+ /// \return The cost of a shuffle instruction of kind Kind with inputs of type
+ /// SrcTy, producing a vector of type DstTy. The exact mask may be passed as
+ /// Mask, or else the array will be empty. The index and subtype parameters
+ /// are used by the subvector insertion and extraction shuffle kinds to show
+ /// the insert/extract point and the type of the subvector being
+ /// inserted/extracted. The operands of the shuffle can be passed through \p
+ /// Args, which helps improve the cost estimation in some cases, like in
+ /// broadcast loads.
+ LLVM_ABI InstructionCost
+ getShuffleCost(ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask = {},
+ TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
+ int Index = 0, VectorType *SubTp = nullptr,
+ ArrayRef<const Value *> Args = {},
+ const Instruction *CxtI = nullptr) const;
/// Represents a hint about the context in which a cast is used.
///
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index a80b4c5179bad..b226b6e9e129f 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -711,9 +711,9 @@ class TargetTransformInfoImplBase {
}
virtual InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Ty, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const {
return 1;
}
@@ -1545,13 +1545,14 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
return 0;
if (Shuffle->isExtractSubvectorMask(SubIndex))
- return TargetTTI->getShuffleCost(TTI::SK_ExtractSubvector, VecSrcTy,
- Mask, CostKind, SubIndex, VecTy,
- Operands, Shuffle);
+ return TargetTTI->getShuffleCost(TTI::SK_ExtractSubvector, VecTy,
+ VecSrcTy, Mask, CostKind, SubIndex,
+ VecTy, Operands, Shuffle);
if (Shuffle->isInsertSubvectorMask(NumSubElts, SubIndex))
return TargetTTI->getShuffleCost(
- TTI::SK_InsertSubvector, VecTy, Mask, CostKind, SubIndex,
+ TTI::SK_InsertSubvector, VecTy, VecSrcTy, Mask, CostKind,
+ SubIndex,
FixedVectorType::get(VecTy->getScalarType(), NumSubElts),
Operands, Shuffle);
@@ -1580,21 +1581,24 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
return TargetTTI->getShuffleCost(
IsUnary ? TTI::SK_PermuteSingleSrc : TTI::SK_PermuteTwoSrc, VecTy,
- AdjustMask, CostKind, 0, nullptr, Operands, Shuffle);
+ VecTy, AdjustMask, CostKind, 0, nullptr, Operands, Shuffle);
}
// Narrowing shuffle - perform shuffle at original wider width and
// then extract the lower elements.
+ // FIXME: This can assume widening, which is not true of all vector
+ // architectures (and is not even the default).
AdjustMask.append(NumSubElts - Mask.size(), PoisonMaskElem);
InstructionCost ShuffleCost = TargetTTI->getShuffleCost(
IsUnary ? TTI::SK_PermuteSingleSrc : TTI::SK_PermuteTwoSrc,
- VecSrcTy, AdjustMask, CostKind, 0, nullptr, Operands, Shuffle);
+ VecSrcTy, VecSrcTy, AdjustMask, CostKind, 0, nullptr, Operands,
+ Shuffle);
SmallVector<int, 16> ExtractMask(Mask.size());
std::iota(ExtractMask.begin(), ExtractMask.end(), 0);
return ShuffleCost + TargetTTI->getShuffleCost(
- TTI::SK_ExtractSubvector, VecSrcTy,
+ TTI::SK_ExtractSubvector, VecTy, VecSrcTy,
ExtractMask, CostKind, 0, VecTy, {}, Shuffle);
}
@@ -1602,40 +1606,44 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
return 0;
if (Shuffle->isReverse())
- return TargetTTI->getShuffleCost(TTI::SK_Reverse, VecTy, Mask, CostKind,
- 0, nullptr, Operands, Shuffle);
+ return TargetTTI->getShuffleCost(TTI::SK_Reverse, VecTy, VecSrcTy, Mask,
+ CostKind, 0, nullptr, Operands,
+ Shuffle);
if (Shuffle->isSelect())
- return TargetTTI->getShuffleCost(TTI::SK_Select, VecTy, Mask, CostKind,
- 0, nullptr, Operands, Shuffle);
+ return TargetTTI->getShuffleCost(TTI::SK_Select, VecTy, VecSrcTy, Mask,
+ CostKind, 0, nullptr, Operands,
+ Shuffle);
if (Shuffle->isTranspose())
- return TargetTTI->getShuffleCost(TTI::SK_Transpose, VecTy, Mask,
- CostKind, 0, nullptr, Operands,
+ return TargetTTI->getShuffleCost(TTI::SK_Transpose, VecTy, VecSrcTy,
+ Mask, CostKind, 0, nullptr, Operands,
Shuffle);
if (Shuffle->isZeroEltSplat())
- return TargetTTI->getShuffleCost(TTI::SK_Broadcast, VecTy, Mask,
- CostKind, 0, nullptr, Operands,
+ return TargetTTI->getShuffleCost(TTI::SK_Broadcast, VecTy, VecSrcTy,
+ Mask, CostKind, 0, nullptr, Operands,
Shuffle);
if (Shuffle->isSingleSource())
- return TargetTTI->getShuffleCost(TTI::SK_PermuteSingleSrc, VecTy, Mask,
- CostKind, 0, nullptr, Operands,
- Shuffle);
+ return TargetTTI->getShuffleCost(TTI::SK_PermuteSingleSrc, VecTy,
+ VecSrcTy, Mask, CostKind, 0, nullptr,
+ Operands, Shuffle);
if (Shuffle->isInsertSubvectorMask(NumSubElts, SubIndex))
return TargetTTI->getShuffleCost(
- TTI::SK_InsertSubvector, VecTy, Mask, CostKind, SubIndex,
+ TTI::SK_InsertSubvector, VecTy, VecSrcTy, Mask, CostKind, SubIndex,
FixedVectorType::get(VecTy->getScalarType(), NumSubElts), Operands,
Shuffle);
if (Shuffle->isSplice(SubIndex))
- return TargetTTI->getShuffleCost(TTI::SK_Splice, VecTy, Mask, CostKind,
- SubIndex, nullptr, Operands, Shuffle);
+ return TargetTTI->getShuffleCost(TTI::SK_Splice, VecTy, VecSrcTy, Mask,
+ CostKind, SubIndex, nullptr, Operands,
+ Shuffle);
- return TargetTTI->getShuffleCost(TTI::SK_PermuteTwoSrc, VecTy, Mask,
- CostKind, 0, nullptr, Operands, Shuffle);
+ return TargetTTI->getShuffleCost(TTI::SK_PermuteTwoSrc, VecTy, VecSrcTy,
+ Mask, CostKind, 0, nullptr, Operands,
+ Shuffle);
}
case Instruction::ExtractElement: {
auto *EEI = dyn_cast<ExtractElementInst>(U);
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index 574152e254f15..b810d93361486 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -329,11 +329,11 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
// Cost the call + mask.
auto Cost =
thisT()->getCallInstrCost(nullptr, RetTy, ICA.getArgTypes(), CostKind);
- if (VD->isMasked())
- Cost += thisT()->getShuffleCost(
- TargetTransformInfo::SK_Broadcast,
- VectorType::get(IntegerType::getInt1Ty(Ctx), VF), {}, CostKind, 0,
- nullptr, {});
+ if (VD->isMasked()) {
+ auto VecTy = VectorType::get(IntegerType::getInt1Ty(Ctx), VF);
+ Cost += thisT()->getShuffleCost(TargetTransformInfo::SK_Broadcast, VecTy,
+ VecTy, {}, CostKind, 0, nullptr, {});
+ }
// Lowering to a library call (with output pointers) may require us to emit
// reloads for the results.
@@ -1101,11 +1101,11 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
TTI::ShuffleKind improveShuffleKindFromMask(TTI::ShuffleKind Kind,
ArrayRef<int> Mask,
- VectorType *Ty, int &Index,
+ VectorType *SrcTy, int &Index,
VectorType *&SubTy) const {
if (Mask.empty())
return Kind;
- int NumSrcElts = Ty->getElementCount().getKnownMinValue();
+ int NumSrcElts = SrcTy->getElementCount().getKnownMinValue();
switch (Kind) {
case TTI::SK_PermuteSingleSrc: {
if (ShuffleVectorInst::isReverseMask(Mask, NumSrcElts))
@@ -1116,7 +1116,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
return TTI::SK_Broadcast;
if (ShuffleVectorInst::isExtractSubvectorMask(Mask, NumSrcElts, Index) &&
(Index + Mask.size()) <= (size_t)NumSrcElts) {
- SubTy = FixedVectorType::get(Ty->getElementType(), Mask.size());
+ SubTy = FixedVectorType::get(SrcTy->getElementType(), Mask.size());
return TTI::SK_ExtractSubvector;
}
break;
@@ -1127,7 +1127,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
Mask, NumSrcElts, NumSubElts, Index)) {
if (Index + NumSubElts > NumSrcElts)
return Kind;
- SubTy = FixedVectorType::get(Ty->getElementType(), NumSubElts);
+ SubTy = FixedVectorType::get(SrcTy->getElementType(), NumSubElts);
return TTI::SK_InsertSubvector;
}
if (ShuffleVectorInst::isSelectMask(Mask, NumSrcElts))
@@ -1151,13 +1151,13 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
}
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override {
- switch (improveShuffleKindFromMask(Kind, Mask, Tp, Index, SubTp)) {
+ switch (improveShuffleKindFromMask(Kind, Mask, SrcTy, Index, SubTp)) {
case TTI::SK_Broadcast:
- if (auto *FVT = dyn_cast<FixedVectorType>(Tp))
+ if (auto *FVT = dyn_cast<FixedVectorType>(SrcTy))
return getBroadcastShuffleOverhead(FVT, CostKind);
return InstructionCost::getInvalid();
case TTI::SK_Select:
@@ -1166,14 +1166,14 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
case TTI::SK_Transpose:
case TTI::SK_PermuteSingleSrc:
case TTI::SK_PermuteTwoSrc:
- if (auto *FVT = dyn_cast<FixedVectorType>(Tp))
+ if (auto *FVT = dyn_cast<FixedVectorType>(SrcTy))
return getPermuteShuffleOverhead(FVT, CostKind);
return InstructionCost::getInvalid();
case TTI::SK_ExtractSubvector:
- return getExtractSubvectorOverhead(Tp, CostKind, Index,
+ return getExtractSubvectorOverhead(SrcTy, CostKind, Index,
cast<FixedVectorType>(SubTp));
case TTI::SK_InsertSubvector:
- return getInsertSubvectorOverhead(Tp, CostKind, Index,
+ return getInsertSubvectorOverhead(DstTy, CostKind, Index,
cast<FixedVectorType>(SubTp));
}
llvm_unreachable("Unknown TTI::ShuffleKind");
@@ -1910,6 +1910,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
return BaseT::getIntrinsicInstrCost(ICA, CostKind);
unsigned Index = cast<ConstantInt>(Args[1])->getZExtValue();
return thisT()->getShuffleCost(TTI::SK_ExtractSubvector,
+ cast<VectorType>(RetTy),
cast<VectorType>(Args[0]->getType()), {},
CostKind, Index, cast<VectorType>(RetTy));
}
@@ -1920,17 +1921,18 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
return BaseT::getIntrinsicInstrCost(ICA, CostKind);
unsigned Index = cast<ConstantInt>(Args[2])->getZExtValue();
return thisT()->getShuffleCost(
- TTI::SK_InsertSubvector, cast<VectorType>(Args[0]->getType()), {},
- CostKind, Index, cast<VectorType>(Args[1]->getType()));
+ TTI::SK_InsertSubvector, cast<VectorType>(RetTy),
+ cast<VectorType>(Args[0]->getType()), {}, CostKind, Index,
+ cast<VectorType>(Args[1]->getType()));
}
case Intrinsic::vector_reverse: {
- return thisT()->getShuffleCost(TTI::SK_Reverse,
+ return thisT()->getShuffleCost(TTI::SK_Reverse, cast<VectorType>(RetTy),
cast<VectorType>(Args[0]->getType()), {},
CostKind, 0, cast<VectorType>(RetTy));
}
case Intrinsic::vector_splice: {
unsigned Index = cast<ConstantInt>(Args[2])->getZExtValue();
- return thisT()->getShuffleCost(TTI::SK_Splice,
+ return thisT()->getShuffleCost(TTI::SK_Splice, cast<VectorType>(RetTy),
cast<VectorType>(Args[0]->getType()), {},
CostKind, Index, cast<VectorType>(RetTy));
}
@@ -2376,8 +2378,8 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
CostKind, 1, nullptr, nullptr);
Cost += thisT()->getVectorInstrCost(Instruction::InsertElement, SearchTy,
CostKind, 0, nullptr, nullptr);
- Cost += thisT()->getShuffleCost(TTI::SK_Broadcast, SearchTy, std::nullopt,
- CostKind, 0, nullptr);
+ Cost += thisT()->getShuffleCost(TTI::SK_Broadcast, SearchTy, SearchTy,
+ std::nullopt, CostKind, 0, nullptr);
Cost += thisT()->getCmpSelInstrCost(BinaryOperator::ICmp, SearchTy, RetTy,
CmpInst::ICMP_EQ, CostKind);
Cost +=
@@ -2961,8 +2963,8 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
while (NumVecElts > MVTLen) {
NumVecElts /= 2;
VectorType *SubTy = FixedVectorType::get(ScalarTy, NumVecElts);
- ShuffleCost += thisT()->getShuffleCost(TTI::SK_ExtractSubvector, Ty, {},
- CostKind, NumVecElts, SubTy);
+ ShuffleCost += thisT()->getShuffleCost(
+ TTI::SK_ExtractSubvector, SubTy, Ty, {}, CostKind, NumVecElts, SubTy);
ArithCost += thisT()->getArithmeticInstrCost(Opcode, SubTy, CostKind);
Ty = SubTy;
++LongVectorCount;
@@ -2978,7 +2980,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
// By default reductions need one shuffle per reduction level.
ShuffleCost +=
NumReduxLevels * thisT()->getShuffleCost(TTI::SK_PermuteSingleSrc, Ty,
- {}, CostKind, 0, Ty);
+ Ty, {}, CostKind, 0, Ty);
ArithCost +=
NumReduxLevels * thisT()->getArithmeticInstrCost(Opcode, Ty, CostKind);
return ShuffleCost + ArithCost +
@@ -3052,8 +3054,8 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
NumVecElts /= 2;
auto *SubTy = FixedVectorType::get(ScalarTy, NumVecElts);
- ShuffleCost += thisT()->getShuffleCost(TTI::SK_ExtractSubvector, Ty, {},
- CostKind, NumVecElts, SubTy);
+ ShuffleCost += thisT()->getShuffleCost(
+ TTI::SK_ExtractSubvector, SubTy, Ty, {}, CostKind, NumVecElts, SubTy);
IntrinsicCostAttributes Attrs(IID, SubTy, {SubTy, SubTy}, FMF);
MinMaxCost += getIntrinsicInstrCost(Attrs, CostKind);
@@ -3069,7 +3071,7 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
// architecture-dependent length.
ShuffleCost +=
NumReduxLevels * thisT()->getShuffleCost(TTI::SK_PermuteSingleSrc, Ty,
- {}, CostKind, 0, Ty);
+ Ty, {}, CostKind, 0, Ty);
IntrinsicCostAttributes Attrs(IID, Ty, {Ty, Ty}, FMF);
MinMaxCost += NumReduxLevels * getIntrinsicInstrCost(Attrs, CostKind);
// The last min/max should be in vector registers and we counted it above.
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index 2d053e55bdfa9..e416860b4974a 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -984,11 +984,16 @@ InstructionCost TargetTransformInfo::getAltInstrCost(
}
InstructionCost TargetTransformInfo::getShuffleCost(
- ShuffleKind Kind, VectorType *Ty, ArrayRef<int> Mask,
+ ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy, ArrayRef<int> Mask,
TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
ArrayRef<const Value *> Args, const Instruction *CxtI) const {
- InstructionCost Cost = TTIImpl->getShuffleCost(Kind, Ty, Mask, CostKind,
- Index, SubTp, Args, CxtI);
+ assert((Mask.empty() || DstTy->isScalableTy() ||
+ Mask.size() == DstTy->getElementCount().getKnownMinValue()) &&
+ "Expected the Mask to match the return size if given");
+ assert(SrcTy->getScalarType() == DstTy->getScalarType() &&
+ "Expected the same scalar types");
+ InstructionCost Cost = TTIImpl->getShuffleCost(
+ Kind, DstTy, SrcTy, Mask, CostKind, Index, SubTp, Args, CxtI);
assert(Cost >= 0 && "TTI should not produce negative costs!");
return Cost;
}
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
index 68aec80f07e1d..ad582d45e77da 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
@@ -5439,19 +5439,25 @@ InstructionCost AArch64TTIImpl::getPartialReductionCost(
return Cost;
}
-InstructionCost AArch64TTIImpl::getShuffleCost(
- TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args, const Instruction *CxtI) const {
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+InstructionCost
+AArch64TTIImpl::getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy,
+ VectorType *SrcTy, ArrayRef<int> Mask,
+ TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args,
+ const Instruction *CxtI) const {
+ assert((Mask.empty() || DstTy->isScalableTy() ||
+ Mask.size() == DstTy->getElementCount().getKnownMinValue()) &&
+ "Expected the Mask to match the return size if given");
+ assert(SrcTy->getScalarType() == DstTy->getScalarType() &&
+ "Expected the same scalar types");
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
// If we have a Mask, and the LT is being legalized somehow, split the Mask
// into smaller vectors and sum the cost of each shuffle.
- if (!Mask.empty() && isa<FixedVectorType>(Tp) && LT.second.isVector() &&
+ if (!Mask.empty() && isa<FixedVectorType>(SrcTy) && LT.second.isVector() &&
LT.second.getScalarSizeInBits() * Mask.size() > 128 &&
- Tp->getScalarSizeInBits() == LT.second.getScalarSizeInBits() &&
+ SrcTy->getScalarSizeInBits() == LT.second.getScalarSizeInBits() &&
Mask.size() > LT.second.getVectorNumElements() && !Index && !SubTp) {
-
// Check for LD3/LD4 instructions, which are represented in llvm IR as
// deinterleaving-shuffle(load). The shuffle cost could potentially be free,
// but we model it with a cost of LT.first so that LD3/LD4 have a higher
@@ -5467,16 +5473,16 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
// cost than just the store.
if (CxtI && CxtI->hasOneUse() && isa<StoreInst>(*CxtI->user_begin()) &&
(ShuffleVectorInst::isInterleaveMask(
- Mask, 4, Tp->getElementCount().getKnownMinValue() * 2) ||
+ Mask, 4, SrcTy->getElementCount().getKnownMinValue() * 2) ||
ShuffleVectorInst::isInterleaveMask(
- Mask, 3, Tp->getElementCount().getKnownMinValue() * 2)))
+ Mask, 3, SrcTy->getElementCount().getKnownMinValue() * 2)))
return LT.first;
unsigned TpNumElts = Mask.size();
unsigned LTNumElts = LT.second.getVectorNumElements();
unsigned NumVecs = (TpNumElts + LTNumElts - 1) / LTNumElts;
- VectorType *NTp =
- VectorType::get(Tp->getScalarType(), LT.second.getVectorElementCount());
+ VectorType *NTp = VectorType::get(SrcTy->getScalarType(),
+ LT.second.getVectorElementCount());
InstructionCost Cost;
std::map<std::tuple<unsigned, unsigned, SmallVector<int>>, InstructionCost>
PreviousCosts;
@@ -5484,7 +5490,7 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
SmallVector<int> NMask;
// Split the existing mask into chunks of size LTNumElts. Track the source
// sub-vectors to ensure the result has at most 2 inputs.
- unsigned Source1 = 0, Source2 = 0;
+ unsigned Source1 = -1U, Source2 = -1U;
unsigned NumSources = 0;
for (unsigned E = 0; E < LTNumElts; E++) {
int MaskElt = (N * LTNumElts + E < TpNumElts) ? Mask[N * LTNumElts + E]
@@ -5532,7 +5538,7 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
NumSources <= 2
? getShuffleCost(NumSources <= 1 ? TTI::SK_PermuteSingleSrc
: TTI::SK_PermuteTwoSrc,
- NTp, NMask, CostKind, 0, nullptr, Args, CxtI)
+ NTp, NTp, NMask, CostKind, 0, nullptr, Args, CxtI)
: LTNumElts;
Result.first->second = NCost;
Cost += NCost;
@@ -5540,7 +5546,7 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
return Cost;
}
- Kind = improveShuffleKindFromMask(Kind, Mask, Tp, Index, SubTp);
+ Kind = improveShuffleKindFromMask(Kind, Mask, SrcTy, Index, SubTp);
bool IsExtractSubvector = Kind == TTI::SK_ExtractSubvector;
// A subvector extract can be implemented with an ext (or trivial extract, if
// from lane 0). This currently only handles low or high extracts to prevent
@@ -5556,6 +5562,12 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
}
Kind = TTI::SK_PermuteSingleSrc;
}
+ // FIXME: This was added to keep the costs equal when adding DstTys. Update
+ // the code to handle length-changing shuffles.
+ if (Kind == TTI::SK_InsertSubvector) {
+ LT = getTypeLegalizationCost(DstTy);
+ SrcTy = DstTy;
+ }
// Check for broadcast loads, which are supported by the LD1R instruction.
// In terms of code-size, the shuffle vector is free when a load + dup get
@@ -5567,15 +5579,17 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
if (CostKind == TTI::TCK_CodeSize && Kind == TTI::SK_Broadcast) {
bool IsLoad = !Args.empty() && isa<LoadInst>(Args[0]);
if (IsLoad && LT.second.isVector() &&
- isLegalBroadcastLoad(Tp->getElementType(),
+ isLegalBroadcastLoad(SrcTy->getElementType(),
LT.second.getVectorElementCount()))
return 0;
}
// If we have 4 elements for the shuffle and a Mask, get the cost straight
// from the perfect shuffle tables.
- if (Mask.size() == 4 && Tp->getElementCount() == ElementCount::getFixed(4) &&
- (Tp->getScalarSizeInBits() == 16 || Tp->getScalarSizeInBits() == 32) &&
+ if (Mask.size() == 4 &&
+ SrcTy->getElementCount() == ElementCount::getFixed(4) &&
+ (SrcTy->getScalarSizeInBits() == 16 ||
+ SrcTy->getScalarSizeInBits() == 32) &&
all_of(Mask, [](int E) { return E < 8; }))
return getPerfectShuffleCost(Mask);
@@ -5735,8 +5749,8 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
return LT.first * Entry->Cost;
}
- if (Kind == TTI::SK_Splice && isa<ScalableVectorType>(Tp))
- return getSpliceCost(Tp, Index, CostKind);
+ if (Kind == TTI::SK_Splice && isa<ScalableVectorType>(SrcTy))
+ return getSpliceCost(SrcTy, Index, CostKind);
// Inserting a subvector can often be done with either a D, S or H register
// move, so long as the inserted vector is "aligned".
@@ -5754,8 +5768,8 @@ InstructionCost AArch64TTIImpl::getShuffleCost(
// Restore optimal kind.
if (IsExtractSubvector)
Kind = TTI::SK_ExtractSubvector;
- return BaseT::getShuffleCost(Kind, Tp, Mask, CostKind, Index, SubTp, Args,
- CxtI);
+ return BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind, Index, SubTp,
+ Args, CxtI);
}
static bool containsDecreasingPointers(Loop *TheLoop,
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h
index 664c360032ea3..799f44a137ce9 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.h
@@ -452,9 +452,9 @@ class AArch64TTIImpl : public BasicTTIImplBase<AArch64TTIImpl> {
TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput) const override;
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
InstructionCost getScalarizationOverhead(
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
index 86a6e49fce027..428ab352cc996 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
@@ -1141,21 +1141,23 @@ Value *GCNTTIImpl::rewriteIntrinsicWithAddressSpace(IntrinsicInst *II,
}
InstructionCost GCNTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
- VectorType *VT, ArrayRef<int> Mask,
+ VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask,
TTI::TargetCostKind CostKind,
int Index, VectorType *SubTp,
ArrayRef<const Value *> Args,
const Instruction *CxtI) const {
- if (!isa<FixedVectorType>(VT))
- return BaseT::getShuffleCost(Kind, VT, Mask, CostKind, Index, SubTp);
+ if (!isa<FixedVectorType>(SrcTy))
+ return BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind, Index,
+ SubTp);
- Kind = improveShuffleKindFromMask(Kind, Mask, VT, Index, SubTp);
+ Kind = improveShuffleKindFromMask(Kind, Mask, SrcTy, Index, SubTp);
// Larger vector widths may require additional instructions, but are
// typically cheaper than scalarized versions.
- unsigned NumVectorElts = cast<FixedVectorType>(VT)->getNumElements();
+ unsigned NumVectorElts = cast<FixedVectorType>(SrcTy)->getNumElements();
if (ST->getGeneration() >= AMDGPUSubtarget::VOLCANIC_ISLANDS &&
- DL.getTypeSizeInBits(VT->getElementType()) == 16) {
+ DL.getTypeSizeInBits(SrcTy->getElementType()) == 16) {
bool HasVOP3P = ST->hasVOP3PInsts();
unsigned RequestedElts =
count_if(Mask, [](int MaskElt) { return MaskElt != -1; });
@@ -1197,7 +1199,8 @@ InstructionCost GCNTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
}
}
- return BaseT::getShuffleCost(Kind, VT, Mask, CostKind, Index, SubTp);
+ return BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind, Index,
+ SubTp);
}
/// Whether it is profitable to sink the operands of an
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h
index ec298c7e9631a..b6ce8fd16a9c8 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h
@@ -236,9 +236,9 @@ class GCNTTIImpl final : public BasicTTIImplBase<GCNTTIImpl> {
InstructionCost getVectorSplitCost() const { return 0; }
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
bool isProfitableToSinkOperands(Instruction *I,
diff --git a/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp b/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
index 6c3a1ae7e1775..203fb76d7be86 100644
--- a/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
+++ b/llvm/lib/Target/ARM/ARMTargetTransformInfo.cpp
@@ -1233,12 +1233,19 @@ InstructionCost ARMTTIImpl::getMemcpyCost(const Instruction *I) const {
}
InstructionCost ARMTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
- VectorType *Tp, ArrayRef<int> Mask,
+ VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask,
TTI::TargetCostKind CostKind,
int Index, VectorType *SubTp,
ArrayRef<const Value *> Args,
const Instruction *CxtI) const {
- Kind = improveShuffleKindFromMask(Kind, Mask, Tp, Index, SubTp);
+ assert((Mask.empty() || DstTy->isScalableTy() ||
+ Mask.size() == DstTy->getElementCount().getKnownMinValue()) &&
+ "Expected the Mask to match the return size if given");
+ assert(SrcTy->getScalarType() == DstTy->getScalarType() &&
+ "Expected the same scalar types");
+
+ Kind = improveShuffleKindFromMask(Kind, Mask, SrcTy, Index, SubTp);
// Treat extractsubvector as single op permutation.
bool IsExtractSubvector = Kind == TTI::SK_ExtractSubvector;
if (IsExtractSubvector)
@@ -1259,7 +1266,7 @@ InstructionCost ARMTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
{ISD::VECTOR_SHUFFLE, MVT::v8i16, 1},
{ISD::VECTOR_SHUFFLE, MVT::v16i8, 1}};
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
if (const auto *Entry =
CostTableLookup(NEONDupTbl, ISD::VECTOR_SHUFFLE, LT.second))
return LT.first * Entry->Cost;
@@ -1280,7 +1287,7 @@ InstructionCost ARMTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
{ISD::VECTOR_SHUFFLE, MVT::v8i16, 2},
{ISD::VECTOR_SHUFFLE, MVT::v16i8, 2}};
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
if (const auto *Entry =
CostTableLookup(NEONShuffleTbl, ISD::VECTOR_SHUFFLE, LT.second))
return LT.first * Entry->Cost;
@@ -1304,7 +1311,7 @@ InstructionCost ARMTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
{ISD::VECTOR_SHUFFLE, MVT::v16i8, 32}};
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
if (const auto *Entry = CostTableLookup(NEONSelShuffleTbl,
ISD::VECTOR_SHUFFLE, LT.second))
return LT.first * Entry->Cost;
@@ -1320,7 +1327,7 @@ InstructionCost ARMTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
{ISD::VECTOR_SHUFFLE, MVT::v4f32, 1},
{ISD::VECTOR_SHUFFLE, MVT::v8f16, 1}};
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
if (const auto *Entry = CostTableLookup(MVEDupTbl, ISD::VECTOR_SHUFFLE,
LT.second))
return LT.first * Entry->Cost *
@@ -1328,7 +1335,7 @@ InstructionCost ARMTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
}
if (!Mask.empty()) {
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
if (LT.second.isVector() &&
Mask.size() <= LT.second.getVectorNumElements() &&
(isVREVMask(Mask, LT.second, 16) || isVREVMask(Mask, LT.second, 32) ||
@@ -1340,11 +1347,11 @@ InstructionCost ARMTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
// Restore optimal kind.
if (IsExtractSubvector)
Kind = TTI::SK_ExtractSubvector;
- int BaseCost = ST->hasMVEIntegerOps() && Tp->isVectorTy()
+ int BaseCost = ST->hasMVEIntegerOps() && SrcTy->isVectorTy()
? ST->getMVEVectorCostFactor(TTI::TCK_RecipThroughput)
: 1;
- return BaseCost *
- BaseT::getShuffleCost(Kind, Tp, Mask, CostKind, Index, SubTp);
+ return BaseCost * BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind,
+ Index, SubTp);
}
InstructionCost ARMTTIImpl::getArithmeticInstrCost(
diff --git a/llvm/lib/Target/ARM/ARMTargetTransformInfo.h b/llvm/lib/Target/ARM/ARMTargetTransformInfo.h
index 20a2c59511087..521f6f08c1361 100644
--- a/llvm/lib/Target/ARM/ARMTargetTransformInfo.h
+++ b/llvm/lib/Target/ARM/ARMTargetTransformInfo.h
@@ -223,9 +223,9 @@ class ARMTTIImpl : public BasicTTIImplBase<ARMTTIImpl> {
int getNumMemOps(const IntrinsicInst *I) const;
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
bool preferInLoopReduction(RecurKind Kind, Type *Ty) const override;
diff --git a/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.cpp b/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.cpp
index a4cc472fdbf29..9fb7d471fd22a 100644
--- a/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.cpp
+++ b/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.cpp
@@ -226,10 +226,12 @@ HexagonTTIImpl::getMaskedMemoryOpCost(unsigned Opcode, Type *Src,
CostKind);
}
-InstructionCost HexagonTTIImpl::getShuffleCost(
- TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args, const Instruction *CxtI) const {
+InstructionCost
+HexagonTTIImpl::getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy,
+ VectorType *SrcTy, ArrayRef<int> Mask,
+ TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args,
+ const Instruction *CxtI) const {
return 1;
}
diff --git a/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.h b/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.h
index d7509c3bb1d2f..a5d07d590006f 100644
--- a/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.h
+++ b/llvm/lib/Target/Hexagon/HexagonTargetTransformInfo.h
@@ -123,9 +123,9 @@ class HexagonTTIImpl : public BasicTTIImplBase<HexagonTTIImpl> {
unsigned AddressSpace,
TTI::TargetCostKind CostKind) const override;
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
InstructionCost getGatherScatterOpCost(unsigned Opcode, Type *DataTy,
const Value *Ptr, bool VariableMask,
diff --git a/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp b/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp
index f9e77f2abdca2..4db983073dcf4 100644
--- a/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp
+++ b/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.cpp
@@ -604,19 +604,20 @@ InstructionCost PPCTTIImpl::getArithmeticInstrCost(
}
InstructionCost PPCTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
- VectorType *Tp, ArrayRef<int> Mask,
+ VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask,
TTI::TargetCostKind CostKind,
int Index, VectorType *SubTp,
ArrayRef<const Value *> Args,
const Instruction *CxtI) const {
InstructionCost CostFactor =
- vectorCostAdjustmentFactor(Instruction::ShuffleVector, Tp, nullptr);
+ vectorCostAdjustmentFactor(Instruction::ShuffleVector, SrcTy, nullptr);
if (!CostFactor.isValid())
return InstructionCost::getMax();
// Legalize the type.
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
// PPC, for both Altivec/VSX, support cheap arbitrary permutations
// (at least in the sense that there need only be one non-loop-invariant
diff --git a/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h b/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h
index 361b2ff223ea0..b7898354df12e 100644
--- a/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h
+++ b/llvm/lib/Target/PowerPC/PPCTargetTransformInfo.h
@@ -111,9 +111,9 @@ class PPCTTIImpl : public BasicTTIImplBase<PPCTTIImpl> {
ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
InstructionCost
getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 860d787111ce4..b187bbefff37e 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -454,12 +454,16 @@ static InstructionCost costShuffleViaSplitting(const RISCVTTIImpl &TTI,
if (!ReusedSingleSrcShuffles.insert(std::make_pair(RegMask, SrcReg))
.second)
return;
- Cost += TTI.getShuffleCost(TTI::SK_PermuteSingleSrc, SingleOpTy,
- RegMask, CostKind, 0, nullptr);
+ Cost += TTI.getShuffleCost(
+ TTI::SK_PermuteSingleSrc,
+ FixedVectorType::get(SingleOpTy->getElementType(), RegMask.size()),
+ SingleOpTy, RegMask, CostKind, 0, nullptr);
},
[&](ArrayRef<int> RegMask, unsigned Idx1, unsigned Idx2, bool NewReg) {
- Cost += TTI.getShuffleCost(TTI::SK_PermuteTwoSrc, SingleOpTy, RegMask,
- CostKind, 0, nullptr);
+ Cost += TTI.getShuffleCost(
+ TTI::SK_PermuteTwoSrc,
+ FixedVectorType::get(SingleOpTy->getElementType(), RegMask.size()),
+ SingleOpTy, RegMask, CostKind, 0, nullptr);
});
return Cost;
}
@@ -528,11 +532,11 @@ costShuffleViaVRegSplitting(const RISCVTTIImpl &TTI, MVT LegalVT,
return;
++NumShuffles;
Cost += TTI.getShuffleCost(TTI::SK_PermuteSingleSrc, SingleOpTy,
- RegMask, CostKind, 0, nullptr);
+ SingleOpTy, RegMask, CostKind, 0, nullptr);
},
[&](ArrayRef<int> RegMask, unsigned Idx1, unsigned Idx2, bool NewReg) {
- Cost += TTI.getShuffleCost(TTI::SK_PermuteTwoSrc, SingleOpTy, RegMask,
- CostKind, 0, nullptr);
+ Cost += TTI.getShuffleCost(TTI::SK_PermuteTwoSrc, SingleOpTy,
+ SingleOpTy, RegMask, CostKind, 0, nullptr);
NumShuffles += 2;
});
// Note: check that we do not emit too many shuffles here to prevent code
@@ -603,22 +607,29 @@ InstructionCost RISCVTTIImpl::getSlideCost(FixedVectorType *Tp,
return FirstSlideCost + SecondSlideCost + MaskCost;
}
-InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
- VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind,
- int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args,
- const Instruction *CxtI) const {
- Kind = improveShuffleKindFromMask(Kind, Mask, Tp, Index, SubTp);
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(Tp);
+InstructionCost
+RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy,
+ VectorType *SrcTy, ArrayRef<int> Mask,
+ TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args,
+ const Instruction *CxtI) const {
+ assert((Mask.empty() || DstTy->isScalableTy() ||
+ Mask.size() == DstTy->getElementCount().getKnownMinValue()) &&
+ "Expected the Mask to match the return size if given");
+ assert(SrcTy->getScalarType() == DstTy->getScalarType() &&
+ "Expected the same scalar types");
+
+ Kind = improveShuffleKindFromMask(Kind, Mask, SrcTy, Index, SubTp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
// First, handle cases where having a fixed length vector enables us to
// give a more accurate cost than falling back to generic scalable codegen.
// TODO: Each of these cases hints at a modeling gap around scalable vectors.
- if (auto *FVTp = dyn_cast<FixedVectorType>(Tp);
+ if (auto *FVTp = dyn_cast<FixedVectorType>(SrcTy);
FVTp && ST->hasVInstructions() && LT.second.isFixedLengthVector()) {
InstructionCost VRegSplittingCost = costShuffleViaVRegSplitting(
- *this, LT.second, ST->getRealVLen(), Tp, Mask, CostKind);
+ *this, LT.second, ST->getRealVLen(),
+ Kind == TTI::SK_InsertSubvector ? DstTy : SrcTy, Mask, CostKind);
if (VRegSplittingCost.isValid())
return VRegSplittingCost;
switch (Kind) {
@@ -657,7 +668,7 @@ InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
for (unsigned I = 0; I != NumSlides; ++I) {
unsigned InsertIndex = SubVectorSize * (1 << I);
FixedVectorType *SubTp =
- FixedVectorType::get(Tp->getElementType(), InsertIndex);
+ FixedVectorType::get(SrcTy->getElementType(), InsertIndex);
FixedVectorType *DestTp =
FixedVectorType::getDoubleElementsVectorType(SubTp);
std::pair<InstructionCost, MVT> DestLT =
@@ -666,7 +677,7 @@ InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
// destination vector register group for vslideup cannot overlap the
// source.
Cost += DestLT.first * TLI->getLMULCost(DestLT.second);
- Cost += getShuffleCost(TTI::SK_InsertSubvector, DestTp, {},
+ Cost += getShuffleCost(TTI::SK_InsertSubvector, DestTp, DestTp, {},
CostKind, InsertIndex, SubTp);
}
return Cost;
@@ -682,7 +693,7 @@ InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
if (LT.first == 1 && (LT.second.getScalarSizeInBits() != 8 ||
LT.second.getVectorNumElements() <= 256)) {
VectorType *IdxTy =
- getVRGatherIndexType(LT.second, *ST, Tp->getContext());
+ getVRGatherIndexType(LT.second, *ST, SrcTy->getContext());
InstructionCost IndexCost = getConstantPoolLoadCost(IdxTy, CostKind);
return IndexCost +
getRISCVInstructionCost(RISCV::VRGATHER_VV, LT.second, CostKind);
@@ -701,8 +712,8 @@ InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
// (shuffle) mask.
if (LT.first == 1 && (LT.second.getScalarSizeInBits() != 8 ||
LT.second.getVectorNumElements() <= 256)) {
- auto &C = Tp->getContext();
- auto EC = Tp->getElementCount();
+ auto &C = SrcTy->getContext();
+ auto EC = SrcTy->getElementCount();
VectorType *IdxTy = getVRGatherIndexType(LT.second, *ST, C);
VectorType *MaskTy = VectorType::get(IntegerType::getInt1Ty(C), EC);
InstructionCost IndexCost = getConstantPoolLoadCost(IdxTy, CostKind);
@@ -771,6 +782,7 @@ InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
// Example sequence:
// vsetivli zero, 4, e8, mf2, tu, ma (ignored)
// vslideup.vi v8, v9, 2
+ LT = getTypeLegalizationCost(DstTy);
return LT.first *
getRISCVInstructionCost(RISCV::VSLIDEUP_VI, LT.second, CostKind);
case TTI::SK_Select: {
@@ -866,11 +878,13 @@ InstructionCost RISCVTTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
InstructionCost GatherCost =
getRISCVInstructionCost(Opcodes, LT.second, CostKind);
// Mask operation additionally required extend and truncate
- InstructionCost ExtendCost = Tp->getElementType()->isIntegerTy(1) ? 3 : 0;
+ InstructionCost ExtendCost =
+ SrcTy->getElementType()->isIntegerTy(1) ? 3 : 0;
return LT.first * (LenCost + GatherCost + ExtendCost);
}
}
- return BaseT::getShuffleCost(Kind, Tp, Mask, CostKind, Index, SubTp);
+ return BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind, Index,
+ SubTp);
}
static unsigned isM1OrSmaller(MVT VT) {
@@ -996,8 +1010,8 @@ InstructionCost RISCVTTIImpl::getInterleavedMemoryOpCost(
auto Mask = createStrideMask(Index, Factor, VF);
Mask.resize(VF * Factor, -1);
InstructionCost ShuffleCost =
- getShuffleCost(TTI::ShuffleKind::SK_PermuteSingleSrc, VecTy, Mask,
- CostKind, 0, nullptr, {});
+ getShuffleCost(TTI::ShuffleKind::SK_PermuteSingleSrc, VecTy, VecTy,
+ Mask, CostKind, 0, nullptr, {});
Cost += ShuffleCost;
}
return Cost;
@@ -1023,7 +1037,7 @@ InstructionCost RISCVTTIImpl::getInterleavedMemoryOpCost(
// shuffle that goes into the wide store
auto Mask = createInterleaveMask(VF, Factor);
InstructionCost ShuffleCost =
- getShuffleCost(TTI::ShuffleKind::SK_PermuteSingleSrc, FVTy, Mask,
+ getShuffleCost(TTI::ShuffleKind::SK_PermuteSingleSrc, FVTy, FVTy, Mask,
CostKind, 0, nullptr, {});
return MemCost + ShuffleCost;
}
@@ -1494,7 +1508,7 @@ RISCVTTIImpl::getIntrinsicInstrCost(const IntrinsicCostAttributes &ICA,
// To support type-based query from vectorizer, set the index to 0.
// Note that index only change the cost from vslide.vx to vslide.vi and in
// current implementations they have same costs.
- return getShuffleCost(TTI::SK_Splice,
+ return getShuffleCost(TTI::SK_Splice, cast<VectorType>(ICA.getReturnType()),
cast<VectorType>(ICA.getArgTypes()[0]), {}, CostKind,
0, cast<VectorType>(ICA.getReturnType()));
}
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
index 0a784461d67bf..23eaee13f4a2a 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.h
@@ -168,9 +168,9 @@ class RISCVTTIImpl : public BasicTTIImplBase<RISCVTTIImpl> {
}
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
InstructionCost getScalarizationOverhead(
diff --git a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
index 68ba7498d586b..f32c9bd2bdea1 100644
--- a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
+++ b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.cpp
@@ -738,20 +738,22 @@ InstructionCost SystemZTTIImpl::getArithmeticInstrCost(
Args, CxtI);
}
-InstructionCost SystemZTTIImpl::getShuffleCost(
- TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args, const Instruction *CxtI) const {
- Kind = improveShuffleKindFromMask(Kind, Mask, Tp, Index, SubTp);
+InstructionCost
+SystemZTTIImpl::getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy,
+ VectorType *SrcTy, ArrayRef<int> Mask,
+ TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args,
+ const Instruction *CxtI) const {
+ Kind = improveShuffleKindFromMask(Kind, Mask, SrcTy, Index, SubTp);
if (ST->hasVector()) {
- unsigned NumVectors = getNumVectorRegs(Tp);
+ unsigned NumVectors = getNumVectorRegs(SrcTy);
// TODO: Since fp32 is expanded, the shuffle cost should always be 0.
// FP128 values are always in scalar registers, so there is no work
// involved with a shuffle, except for broadcast. In that case register
// moves are done with a single instruction per element.
- if (Tp->getScalarType()->isFP128Ty())
+ if (SrcTy->getScalarType()->isFP128Ty())
return (Kind == TargetTransformInfo::SK_Broadcast ? NumVectors - 1 : 0);
switch (Kind) {
@@ -775,7 +777,8 @@ InstructionCost SystemZTTIImpl::getShuffleCost(
}
}
- return BaseT::getShuffleCost(Kind, Tp, Mask, CostKind, Index, SubTp);
+ return BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind, Index,
+ SubTp);
}
// Return the log2 difference of the element sizes of the two vector types.
diff --git a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.h b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.h
index b4bc41974b70b..566244d97fa52 100644
--- a/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.h
+++ b/llvm/lib/Target/SystemZ/SystemZTargetTransformInfo.h
@@ -102,9 +102,9 @@ class SystemZTTIImpl : public BasicTTIImplBase<SystemZTTIImpl> {
ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
unsigned getVectorTruncCost(Type *SrcTy, Type *DstTy) const;
unsigned getVectorBitmaskConversionCost(Type *SrcTy, Type *DstTy) const;
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 9864adc4dcc95..5e001b19f4d8d 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -1522,15 +1522,24 @@ X86TTIImpl::getAltInstrCost(VectorType *VecTy, unsigned Opcode0,
return InstructionCost::getInvalid();
}
-InstructionCost X86TTIImpl::getShuffleCost(
- TTI::ShuffleKind Kind, VectorType *BaseTp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args, const Instruction *CxtI) const {
+InstructionCost X86TTIImpl::getShuffleCost(TTI::ShuffleKind Kind,
+ VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask,
+ TTI::TargetCostKind CostKind,
+ int Index, VectorType *SubTp,
+ ArrayRef<const Value *> Args,
+ const Instruction *CxtI) const {
+ assert((Mask.empty() || DstTy->isScalableTy() ||
+ Mask.size() == DstTy->getElementCount().getKnownMinValue()) &&
+ "Expected the Mask to match the return size if given");
+ assert(SrcTy->getScalarType() == DstTy->getScalarType() &&
+ "Expected the same scalar types");
+
// 64-bit packed float vectors (v2f32) are widened to type v4f32.
// 64-bit packed integer vectors (v2i32) are widened to type v4i32.
- std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(BaseTp);
+ std::pair<InstructionCost, MVT> LT = getTypeLegalizationCost(SrcTy);
- Kind = improveShuffleKindFromMask(Kind, Mask, BaseTp, Index, SubTp);
+ Kind = improveShuffleKindFromMask(Kind, Mask, SrcTy, Index, SubTp);
// If all args are constant than this will be constant folded away.
if (!Args.empty() &&
@@ -1539,11 +1548,12 @@ InstructionCost X86TTIImpl::getShuffleCost(
// Recognize a basic concat_vector shuffle.
if (Kind == TTI::SK_PermuteTwoSrc &&
- Mask.size() == (2 * BaseTp->getElementCount().getKnownMinValue()) &&
+ Mask.size() == (2 * SrcTy->getElementCount().getKnownMinValue()) &&
ShuffleVectorInst::isIdentityMask(Mask, Mask.size()))
return getShuffleCost(TTI::SK_InsertSubvector,
- VectorType::getDoubleElementsVectorType(BaseTp), Mask,
- CostKind, Mask.size() / 2, BaseTp);
+ VectorType::getDoubleElementsVectorType(SrcTy),
+ VectorType::getDoubleElementsVectorType(SrcTy), Mask,
+ CostKind, Mask.size() / 2, SrcTy);
// Treat Transpose as 2-op shuffles - there's no difference in lowering.
if (Kind == TTI::SK_Transpose)
@@ -1568,11 +1578,11 @@ InstructionCost X86TTIImpl::getShuffleCost(
// Attempt to detect a shuffle mask with a single defined element.
bool IsInLaneShuffle = false;
bool IsSingleElementMask = false;
- if (BaseTp->getPrimitiveSizeInBits() > 0 &&
- (BaseTp->getPrimitiveSizeInBits() % 128) == 0 &&
- BaseTp->getScalarSizeInBits() == LT.second.getScalarSizeInBits() &&
- Mask.size() == BaseTp->getElementCount().getKnownMinValue()) {
- unsigned NumLanes = BaseTp->getPrimitiveSizeInBits() / 128;
+ if (SrcTy->getPrimitiveSizeInBits() > 0 &&
+ (SrcTy->getPrimitiveSizeInBits() % 128) == 0 &&
+ SrcTy->getScalarSizeInBits() == LT.second.getScalarSizeInBits() &&
+ Mask.size() == SrcTy->getElementCount().getKnownMinValue()) {
+ unsigned NumLanes = SrcTy->getPrimitiveSizeInBits() / 128;
unsigned NumEltsPerLane = Mask.size() / NumLanes;
if ((Mask.size() % NumLanes) == 0) {
IsInLaneShuffle = all_of(enumerate(Mask), [&](const auto &P) {
@@ -1614,16 +1624,17 @@ InstructionCost X86TTIImpl::getShuffleCost(
LT.second.getVectorElementType() ==
SubLT.second.getVectorElementType() &&
LT.second.getVectorElementType().getSizeInBits() ==
- BaseTp->getElementType()->getPrimitiveSizeInBits()) {
+ SrcTy->getElementType()->getPrimitiveSizeInBits()) {
assert(NumElts >= NumSubElts && NumElts > OrigSubElts &&
"Unexpected number of elements!");
- auto *VecTy = FixedVectorType::get(BaseTp->getElementType(),
+ auto *VecTy = FixedVectorType::get(SrcTy->getElementType(),
LT.second.getVectorNumElements());
- auto *SubTy = FixedVectorType::get(BaseTp->getElementType(),
+ auto *SubTy = FixedVectorType::get(SrcTy->getElementType(),
SubLT.second.getVectorNumElements());
int ExtractIndex = alignDown((Index % NumElts), NumSubElts);
- InstructionCost ExtractCost = getShuffleCost(
- TTI::SK_ExtractSubvector, VecTy, {}, CostKind, ExtractIndex, SubTy);
+ InstructionCost ExtractCost =
+ getShuffleCost(TTI::SK_ExtractSubvector, VecTy, VecTy, {}, CostKind,
+ ExtractIndex, SubTy);
// If the original size is 32-bits or more, we can use pshufd. Otherwise
// if we have SSSE3 we can use pshufb.
@@ -1646,7 +1657,8 @@ InstructionCost X86TTIImpl::getShuffleCost(
// but if the destination vector legalizes to the same width as the subvector
// then the insertion will simplify to a (free) register copy.
if (Kind == TTI::SK_InsertSubvector && LT.second.isVector()) {
- int NumElts = LT.second.getVectorNumElements();
+ std::pair<InstructionCost, MVT> DstLT = getTypeLegalizationCost(DstTy);
+ int NumElts = DstLT.second.getVectorNumElements();
std::pair<InstructionCost, MVT> SubLT = getTypeLegalizationCost(SubTp);
if (SubLT.second.isVector()) {
int NumSubElts = SubLT.second.getVectorNumElements();
@@ -1670,7 +1682,7 @@ InstructionCost X86TTIImpl::getShuffleCost(
// Handle some common (illegal) sub-vector types as they are often very cheap
// to shuffle even on targets without PSHUFB.
- EVT VT = TLI->getValueType(DL, BaseTp);
+ EVT VT = TLI->getValueType(DL, SrcTy);
if (VT.isSimple() && VT.isVector() && VT.getSizeInBits() < 128 &&
!ST->hasSSSE3()) {
static const CostKindTblEntry SSE2SubVectorShuffleTbl[] = {
@@ -1717,17 +1729,17 @@ InstructionCost X86TTIImpl::getShuffleCost(
MVT LegalVT = LT.second;
if (LegalVT.isVector() &&
LegalVT.getVectorElementType().getSizeInBits() ==
- BaseTp->getElementType()->getPrimitiveSizeInBits() &&
+ SrcTy->getElementType()->getPrimitiveSizeInBits() &&
LegalVT.getVectorNumElements() <
- cast<FixedVectorType>(BaseTp)->getNumElements()) {
- unsigned VecTySize = DL.getTypeStoreSize(BaseTp);
+ cast<FixedVectorType>(SrcTy)->getNumElements()) {
+ unsigned VecTySize = DL.getTypeStoreSize(SrcTy);
unsigned LegalVTSize = LegalVT.getStoreSize();
// Number of source vectors after legalization:
unsigned NumOfSrcs = (VecTySize + LegalVTSize - 1) / LegalVTSize;
// Number of destination vectors after legalization:
InstructionCost NumOfDests = LT.first;
- auto *SingleOpTy = FixedVectorType::get(BaseTp->getElementType(),
+ auto *SingleOpTy = FixedVectorType::get(SrcTy->getElementType(),
LegalVT.getVectorNumElements());
if (!Mask.empty() && NumOfDests.isValid()) {
@@ -1746,7 +1758,7 @@ InstructionCost X86TTIImpl::getShuffleCost(
// this operation is TTI::TCC_Free.
NumOfDests =
getTypeLegalizationCost(
- FixedVectorType::get(BaseTp->getElementType(), Mask.size()))
+ FixedVectorType::get(SrcTy->getElementType(), Mask.size()))
.first;
unsigned E = NumOfDests.getValue();
unsigned NormalizedVF =
@@ -1767,8 +1779,9 @@ InstructionCost X86TTIImpl::getShuffleCost(
// one.
if (PrevRegMask.empty() || PrevSrcReg != SrcReg ||
PrevRegMask != RegMask)
- Cost += getShuffleCost(TTI::SK_PermuteSingleSrc, SingleOpTy,
- RegMask, CostKind, 0, nullptr);
+ Cost +=
+ getShuffleCost(TTI::SK_PermuteSingleSrc, SingleOpTy,
+ SingleOpTy, RegMask, CostKind, 0, nullptr);
else
// Just a copy of previous destination register.
Cost += TTI::TCC_Basic;
@@ -1785,18 +1798,20 @@ InstructionCost X86TTIImpl::getShuffleCost(
[this, SingleOpTy, CostKind,
&Cost](ArrayRef<int> RegMask, unsigned /*Unused*/,
unsigned /*Unused*/, bool /*Unused*/) {
- Cost += getShuffleCost(TTI::SK_PermuteTwoSrc, SingleOpTy, RegMask,
- CostKind, 0, nullptr);
+ Cost += getShuffleCost(TTI::SK_PermuteTwoSrc, SingleOpTy,
+ SingleOpTy, RegMask, CostKind, 0, nullptr);
});
return Cost;
}
InstructionCost NumOfShuffles = (NumOfSrcs - 1) * NumOfDests;
return NumOfShuffles * getShuffleCost(TTI::SK_PermuteTwoSrc, SingleOpTy,
- {}, CostKind, 0, nullptr);
+ SingleOpTy, {}, CostKind, 0,
+ nullptr);
}
- return BaseT::getShuffleCost(Kind, BaseTp, Mask, CostKind, Index, SubTp);
+ return BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind, Index,
+ SubTp);
}
// If we're just moving a single element around (probably as an alternative to
@@ -2229,7 +2244,7 @@ InstructionCost X86TTIImpl::getShuffleCost(
if (ST->hasSSE3() && IsLoad)
if (const auto *Entry =
CostTableLookup(SSE3BroadcastLoadTbl, Kind, LT.second)) {
- assert(isLegalBroadcastLoad(BaseTp->getElementType(),
+ assert(isLegalBroadcastLoad(SrcTy->getElementType(),
LT.second.getVectorElementCount()) &&
"Table entry missing from isLegalBroadcastLoad()");
return LT.first * Entry->Cost;
@@ -2263,7 +2278,8 @@ InstructionCost X86TTIImpl::getShuffleCost(
return LT.first * *KindCost;
}
- return BaseT::getShuffleCost(Kind, BaseTp, Mask, CostKind, Index, SubTp);
+ return BaseT::getShuffleCost(Kind, DstTy, SrcTy, Mask, CostKind, Index,
+ SubTp);
}
InstructionCost X86TTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst,
@@ -4903,8 +4919,8 @@ InstructionCost X86TTIImpl::getVectorInstrCost(unsigned Opcode, Type *Val,
EVT VT = TLI->getValueType(DL, Val);
if (VT.getScalarType() != MScalarTy || VT.getSizeInBits() >= 128)
SubTy = FixedVectorType::get(ScalarType, SubNumElts);
- ShuffleCost =
- getShuffleCost(TTI::SK_PermuteTwoSrc, SubTy, {}, CostKind, 0, SubTy);
+ ShuffleCost = getShuffleCost(TTI::SK_PermuteTwoSrc, SubTy, SubTy, {},
+ CostKind, 0, SubTy);
}
int IntOrFpCost = ScalarType->isFloatingPointTy() ? 0 : 1;
return ShuffleCost + IntOrFpCost + RegisterFileMoveCost;
@@ -4999,8 +5015,8 @@ InstructionCost X86TTIImpl::getScalarizationOverhead(
// FIXME: we don't need to extract if all non-demanded elements
// are legalization-inserted padding.
if (!LaneEltMask.isAllOnes())
- Cost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, {}, CostKind,
- I * NumEltsPerLane, LaneTy);
+ Cost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, Ty, {},
+ CostKind, I * NumEltsPerLane, LaneTy);
Cost += BaseT::getScalarizationOverhead(LaneTy, LaneEltMask, Insert,
/*Extract*/ false, CostKind);
}
@@ -5017,8 +5033,8 @@ InstructionCost X86TTIImpl::getScalarizationOverhead(
if (!AffectedLanes[I] ||
(Lane == 0 && FullyAffectedLegalVectors[LegalVec]))
continue;
- Cost += getShuffleCost(TTI::SK_InsertSubvector, Ty, {}, CostKind,
- I * NumEltsPerLane, LaneTy);
+ Cost += getShuffleCost(TTI::SK_InsertSubvector, Ty, Ty, {},
+ CostKind, I * NumEltsPerLane, LaneTy);
}
}
}
@@ -5077,7 +5093,7 @@ InstructionCost X86TTIImpl::getScalarizationOverhead(
NumEltsPerLane, I * NumEltsPerLane);
if (LaneEltMask.isZero())
continue;
- Cost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, {}, CostKind,
+ Cost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, Ty, {}, CostKind,
I * NumEltsPerLane, LaneTy);
Cost += BaseT::getScalarizationOverhead(
LaneTy, LaneEltMask, /*Insert*/ false, Extract, CostKind);
@@ -5195,9 +5211,10 @@ X86TTIImpl::getReplicationShuffleCost(Type *EltTy, int ReplicationFactor,
DemandedDstElts.zext(NumDstVectors * NumEltsPerDstVec), NumDstVectors);
unsigned NumDstVectorsDemanded = DemandedDstVectors.popcount();
- InstructionCost SingleShuffleCost = getShuffleCost(
- TTI::SK_PermuteSingleSrc, SingleDstVecTy, /*Mask=*/{}, CostKind,
- /*Index=*/0, /*SubTp=*/nullptr);
+ InstructionCost SingleShuffleCost =
+ getShuffleCost(TTI::SK_PermuteSingleSrc, SingleDstVecTy, SingleDstVecTy,
+ /*Mask=*/{}, CostKind,
+ /*Index=*/0, /*SubTp=*/nullptr);
return NumDstVectorsDemanded * SingleShuffleCost;
}
@@ -5338,9 +5355,10 @@ InstructionCost X86TTIImpl::getMemoryOpCost(unsigned Opcode, Type *Src,
SubVecEltsLeft += CurrVecTy->getNumElements();
// And that's free only for the 0'th subvector of a legalized vector.
if (!Is0thSubVec)
- Cost += getShuffleCost(IsLoad ? TTI::ShuffleKind::SK_InsertSubvector
- : TTI::ShuffleKind::SK_ExtractSubvector,
- VTy, {}, CostKind, NumEltDone(), CurrVecTy);
+ Cost +=
+ getShuffleCost(IsLoad ? TTI::ShuffleKind::SK_InsertSubvector
+ : TTI::ShuffleKind::SK_ExtractSubvector,
+ VTy, VTy, {}, CostKind, NumEltDone(), CurrVecTy);
}
// While we can directly load/store ZMM, YMM, and 64-bit halves of XMM,
@@ -5416,17 +5434,17 @@ X86TTIImpl::getMaskedMemoryOpCost(unsigned Opcode, Type *SrcTy, Align Alignment,
if (VT.isSimple() && Ty != VT.getSimpleVT() &&
LT.second.getVectorNumElements() == NumElem)
// Promotion requires extend/truncate for data and a shuffle for mask.
- Cost +=
- getShuffleCost(TTI::SK_PermuteTwoSrc, SrcVTy, {}, CostKind, 0,
- nullptr) +
- getShuffleCost(TTI::SK_PermuteTwoSrc, MaskTy, {}, CostKind, 0, nullptr);
+ Cost += getShuffleCost(TTI::SK_PermuteTwoSrc, SrcVTy, SrcVTy, {}, CostKind,
+ 0, nullptr) +
+ getShuffleCost(TTI::SK_PermuteTwoSrc, MaskTy, MaskTy, {}, CostKind,
+ 0, nullptr);
else if (LT.first * Ty.getVectorNumElements() > NumElem) {
auto *NewMaskTy = FixedVectorType::get(MaskTy->getElementType(),
Ty.getVectorNumElements());
// Expanding requires fill mask with zeroes
- Cost += getShuffleCost(TTI::SK_InsertSubvector, NewMaskTy, {}, CostKind, 0,
- MaskTy);
+ Cost += getShuffleCost(TTI::SK_InsertSubvector, NewMaskTy, NewMaskTy, {},
+ CostKind, 0, MaskTy);
}
// Pre-AVX512 - each maskmov load costs 2 + store costs ~8.
@@ -5690,7 +5708,7 @@ X86TTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *ValTy,
// If we're reducing from 256/512 bits, use an extract_subvector.
if (Size > 128) {
auto *SubTy = FixedVectorType::get(ValVTy->getElementType(), NumVecElts);
- ReductionCost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, {},
+ ReductionCost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, Ty, {},
CostKind, NumVecElts, SubTy);
Ty = SubTy;
} else if (Size == 128) {
@@ -5702,8 +5720,8 @@ X86TTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *ValTy,
else
ShufTy =
FixedVectorType::get(Type::getInt64Ty(ValVTy->getContext()), 2);
- ReductionCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, {},
- CostKind, 0, nullptr);
+ ReductionCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, ShufTy,
+ {}, CostKind, 0, nullptr);
} else if (Size == 64) {
// Reducing from 64 bits is a shuffle of v4f32/v4i32.
FixedVectorType *ShufTy;
@@ -5713,8 +5731,8 @@ X86TTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *ValTy,
else
ShufTy =
FixedVectorType::get(Type::getInt32Ty(ValVTy->getContext()), 4);
- ReductionCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, {},
- CostKind, 0, nullptr);
+ ReductionCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, ShufTy,
+ {}, CostKind, 0, nullptr);
} else {
// Reducing from smaller size is a shift by immediate.
auto *ShiftTy = FixedVectorType::get(
@@ -5872,8 +5890,8 @@ X86TTIImpl::getMinMaxReductionCost(Intrinsic::ID IID, VectorType *ValTy,
// If we're reducing from 256/512 bits, use an extract_subvector.
if (Size > 128) {
auto *SubTy = FixedVectorType::get(ValVTy->getElementType(), NumVecElts);
- MinMaxCost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, {}, CostKind,
- NumVecElts, SubTy);
+ MinMaxCost += getShuffleCost(TTI::SK_ExtractSubvector, Ty, Ty, {},
+ CostKind, NumVecElts, SubTy);
Ty = SubTy;
} else if (Size == 128) {
// Reducing from 128 bits is a permute of v2f64/v2i64.
@@ -5883,7 +5901,7 @@ X86TTIImpl::getMinMaxReductionCost(Intrinsic::ID IID, VectorType *ValTy,
FixedVectorType::get(Type::getDoubleTy(ValTy->getContext()), 2);
else
ShufTy = FixedVectorType::get(Type::getInt64Ty(ValTy->getContext()), 2);
- MinMaxCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, {},
+ MinMaxCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, ShufTy, {},
CostKind, 0, nullptr);
} else if (Size == 64) {
// Reducing from 64 bits is a shuffle of v4f32/v4i32.
@@ -5892,7 +5910,7 @@ X86TTIImpl::getMinMaxReductionCost(Intrinsic::ID IID, VectorType *ValTy,
ShufTy = FixedVectorType::get(Type::getFloatTy(ValTy->getContext()), 4);
else
ShufTy = FixedVectorType::get(Type::getInt32Ty(ValTy->getContext()), 4);
- MinMaxCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, {},
+ MinMaxCost += getShuffleCost(TTI::SK_PermuteSingleSrc, ShufTy, ShufTy, {},
CostKind, 0, nullptr);
} else {
// Reducing from smaller size is a shift by immediate.
@@ -6671,8 +6689,8 @@ InstructionCost X86TTIImpl::getInterleavedMemoryOpCostAVX512(
TTI::ShuffleKind ShuffleKind =
(NumOfMemOps > 1) ? TTI::SK_PermuteTwoSrc : TTI::SK_PermuteSingleSrc;
- InstructionCost ShuffleCost =
- getShuffleCost(ShuffleKind, SingleMemOpTy, {}, CostKind, 0, nullptr);
+ InstructionCost ShuffleCost = getShuffleCost(
+ ShuffleKind, SingleMemOpTy, SingleMemOpTy, {}, CostKind, 0, nullptr);
unsigned NumOfLoadsInInterleaveGrp =
Indices.size() ? Indices.size() : Factor;
@@ -6728,8 +6746,9 @@ InstructionCost X86TTIImpl::getInterleavedMemoryOpCostAVX512(
// There is no strided stores meanwhile. And store can't be folded in
// shuffle.
unsigned NumOfSources = Factor; // The number of values to be merged.
- InstructionCost ShuffleCost = getShuffleCost(
- TTI::SK_PermuteTwoSrc, SingleMemOpTy, {}, CostKind, 0, nullptr);
+ InstructionCost ShuffleCost =
+ getShuffleCost(TTI::SK_PermuteTwoSrc, SingleMemOpTy, SingleMemOpTy, {},
+ CostKind, 0, nullptr);
unsigned NumOfShufflesPerStore = NumOfSources - 1;
// The SK_MergeTwoSrc shuffle clobbers one of src operands.
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.h b/llvm/lib/Target/X86/X86TargetTransformInfo.h
index 72673d6fbd80f..e16a88cdfe53c 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.h
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.h
@@ -149,9 +149,9 @@ class X86TTIImpl : public BasicTTIImplBase<X86TTIImpl> {
TTI::TargetCostKind CostKind) const override;
InstructionCost
- getShuffleCost(TTI::ShuffleKind Kind, VectorType *Tp, ArrayRef<int> Mask,
- TTI::TargetCostKind CostKind, int Index, VectorType *SubTp,
- ArrayRef<const Value *> Args = {},
+ getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask, TTI::TargetCostKind CostKind, int Index,
+ VectorType *SubTp, ArrayRef<const Value *> Args = {},
const Instruction *CxtI = nullptr) const override;
InstructionCost
getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
diff --git a/llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp b/llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp
index 756a72e6d97bc..447dea6968903 100644
--- a/llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp
+++ b/llvm/lib/Transforms/Scalar/LowerMatrixIntrinsics.cpp
@@ -1411,9 +1411,9 @@ class LowerMatrixIntrinsics {
InstructionCost EmbedCost(0);
// Roughly estimate the cost for embedding the columns into a vector.
for (unsigned I = 1; I < N; ++I)
- EmbedCost +=
- TTI.getShuffleCost(TTI::SK_Splice, FixedVectorType::get(EltTy, 1),
- {}, TTI::TCK_RecipThroughput);
+ EmbedCost += TTI.getShuffleCost(
+ TTI::SK_Splice, FixedVectorType::get(EltTy, 1),
+ FixedVectorType::get(EltTy, 1), {}, TTI::TCK_RecipThroughput);
return EmbedCost;
}
@@ -1433,9 +1433,9 @@ class LowerMatrixIntrinsics {
// vector.
InstructionCost EmbedCost(0);
for (unsigned I = 1; I < N; ++I)
- EmbedCost -=
- TTI.getShuffleCost(TTI::SK_Splice, FixedVectorType::get(EltTy, 1),
- {}, TTI::TCK_RecipThroughput);
+ EmbedCost -= TTI.getShuffleCost(
+ TTI::SK_Splice, FixedVectorType::get(EltTy, 1),
+ FixedVectorType::get(EltTy, 1), {}, TTI::TCK_RecipThroughput);
return EmbedCost;
}
diff --git a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
index 6954a5f2dab43..5fd44256b3e97 100644
--- a/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
+++ b/llvm/lib/Transforms/Vectorize/LoopVectorize.cpp
@@ -5593,8 +5593,8 @@ LoopVectorizationCostModel::getConsecutiveMemOpCost(Instruction *I,
bool Reverse = ConsecutiveStride < 0;
if (Reverse)
- Cost += TTI.getShuffleCost(TargetTransformInfo::SK_Reverse, VectorTy, {},
- CostKind, 0);
+ Cost += TTI.getShuffleCost(TargetTransformInfo::SK_Reverse, VectorTy,
+ VectorTy, {}, CostKind, 0);
return Cost;
}
@@ -5611,8 +5611,8 @@ LoopVectorizationCostModel::getUniformMemOpCost(Instruction *I,
return TTI.getAddressComputationCost(ValTy) +
TTI.getMemoryOpCost(Instruction::Load, ValTy, Alignment, AS,
CostKind) +
- TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VectorTy, {},
- CostKind);
+ TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VectorTy,
+ VectorTy, {}, CostKind);
}
StoreInst *SI = cast<StoreInst>(I);
@@ -5674,8 +5674,8 @@ LoopVectorizationCostModel::getInterleaveGroupCost(Instruction *I,
assert(!Legal->isMaskRequired(I) &&
"Reverse masked interleaved access not supported.");
Cost += Group->getNumMembers() *
- TTI.getShuffleCost(TargetTransformInfo::SK_Reverse, VectorTy, {},
- CostKind, 0);
+ TTI.getShuffleCost(TargetTransformInfo::SK_Reverse, VectorTy,
+ VectorTy, {}, CostKind, 0);
}
return Cost;
}
@@ -6422,6 +6422,7 @@ LoopVectorizationCostModel::getInstructionCost(Instruction *I,
SmallVector<int> Mask(VF.getKnownMinValue());
std::iota(Mask.begin(), Mask.end(), VF.getKnownMinValue() - 1);
return TTI.getShuffleCost(TargetTransformInfo::SK_Splice,
+ cast<VectorType>(VectorTy),
cast<VectorType>(VectorTy), Mask, CostKind,
VF.getKnownMinValue() - 1);
}
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 3e7f4cb56c0b3..ff2fb67e9563c 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -5693,20 +5693,24 @@ getShuffleCost(const TargetTransformInfo &TTI, TTI::ShuffleKind Kind,
TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
int Index = 0, VectorType *SubTp = nullptr,
ArrayRef<const Value *> Args = {}) {
+ VectorType *DstTy = Tp;
+ if (!Mask.empty())
+ DstTy = FixedVectorType::get(Tp->getScalarType(), Mask.size());
+
if (Kind != TTI::SK_PermuteTwoSrc)
- return TTI.getShuffleCost(Kind, Tp, Mask, CostKind, Index, SubTp, Args);
+ return TTI.getShuffleCost(Kind, DstTy, Tp, Mask, CostKind, Index, SubTp,
+ Args);
int NumSrcElts = Tp->getElementCount().getKnownMinValue();
int NumSubElts;
if (Mask.size() > 2 && ShuffleVectorInst::isInsertSubvectorMask(
Mask, NumSrcElts, NumSubElts, Index)) {
if (Index + NumSubElts > NumSrcElts &&
Index + NumSrcElts <= static_cast<int>(Mask.size()))
- return TTI.getShuffleCost(
- TTI::SK_InsertSubvector,
- getWidenedType(Tp->getElementType(), Mask.size()), Mask,
- TTI::TCK_RecipThroughput, Index, Tp);
+ return TTI.getShuffleCost(TTI::SK_InsertSubvector, DstTy, Tp, Mask,
+ TTI::TCK_RecipThroughput, Index, Tp);
}
- return TTI.getShuffleCost(Kind, Tp, Mask, CostKind, Index, SubTp, Args);
+ return TTI.getShuffleCost(Kind, DstTy, Tp, Mask, CostKind, Index, SubTp,
+ Args);
}
/// This is similar to TargetTransformInfo::getScalarizationOverhead, but if
@@ -12013,7 +12017,7 @@ class BoUpSLP::ShuffleCostEstimator : public BaseShuffleAnalysis {
if (isa<FixedVectorType>(ScalarTy)) {
assert(SLPReVec && "FixedVectorType is not expected.");
return TTI.getShuffleCost(
- TTI::SK_InsertSubvector, VecTy, {}, CostKind,
+ TTI::SK_InsertSubvector, VecTy, VecTy, {}, CostKind,
std::distance(VL.begin(), It) * getNumElements(ScalarTy),
cast<FixedVectorType>(ScalarTy));
}
@@ -22927,7 +22931,10 @@ class HorizontalReduction {
unsigned ScalarTyNumElements = VecTy->getNumElements();
for (unsigned I : seq<unsigned>(ReducedVals.size())) {
VectorCost += TTI->getShuffleCost(
- TTI::SK_PermuteSingleSrc, VectorTy,
+ TTI::SK_PermuteSingleSrc,
+ FixedVectorType::get(VecTy->getScalarType(),
+ ReducedVals.size()),
+ VectorTy,
createStrideMask(I, ScalarTyNumElements, ReducedVals.size()));
VectorCost += TTI->getArithmeticReductionCost(RdxOpcode, VecTy,
FMF, CostKind);
diff --git a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
index d16c8b44891f5..f08787fa41dc2 100644
--- a/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
+++ b/llvm/lib/Transforms/Vectorize/VPlanRecipes.cpp
@@ -809,6 +809,7 @@ InstructionCost VPInstruction::computeCost(ElementCount VF,
Type *VectorTy = toVectorTy(Ctx.Types.inferScalarType(this), VF);
return Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Splice,
+ cast<VectorType>(VectorTy),
cast<VectorType>(VectorTy), Mask,
Ctx.CostKind, VF.getKnownMinValue() - 1);
}
@@ -2971,9 +2972,9 @@ InstructionCost VPWidenMemoryRecipe::computeCost(ElementCount VF,
if (!Reverse)
return Cost;
- return Cost +=
- Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Reverse,
- cast<VectorType>(Ty), {}, Ctx.CostKind, 0);
+ return Cost += Ctx.TTI.getShuffleCost(
+ TargetTransformInfo::SK_Reverse, cast<VectorType>(Ty),
+ cast<VectorType>(Ty), {}, Ctx.CostKind, 0);
}
void VPWidenLoadRecipe::execute(VPTransformState &State) {
@@ -3089,9 +3090,9 @@ InstructionCost VPWidenLoadEVLRecipe::computeCost(ElementCount VF,
if (!Reverse)
return Cost;
- return Cost + Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Reverse,
- cast<VectorType>(Ty), {}, Ctx.CostKind,
- 0);
+ return Cost + Ctx.TTI.getShuffleCost(
+ TargetTransformInfo::SK_Reverse, cast<VectorType>(Ty),
+ cast<VectorType>(Ty), {}, Ctx.CostKind, 0);
}
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
@@ -3204,9 +3205,9 @@ InstructionCost VPWidenStoreEVLRecipe::computeCost(ElementCount VF,
if (!Reverse)
return Cost;
- return Cost + Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Reverse,
- cast<VectorType>(Ty), {}, Ctx.CostKind,
- 0);
+ return Cost + Ctx.TTI.getShuffleCost(
+ TargetTransformInfo::SK_Reverse, cast<VectorType>(Ty),
+ cast<VectorType>(Ty), {}, Ctx.CostKind, 0);
}
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
@@ -3617,8 +3618,8 @@ InstructionCost VPInterleaveRecipe::computeCost(ElementCount VF,
return Cost + IG->getNumMembers() *
Ctx.TTI.getShuffleCost(TargetTransformInfo::SK_Reverse,
- VectorTy, std::nullopt, Ctx.CostKind,
- 0);
+ VectorTy, VectorTy, std::nullopt,
+ Ctx.CostKind, 0);
}
#if !defined(NDEBUG) || defined(LLVM_ENABLE_DUMP)
diff --git a/llvm/lib/Transforms/Vectorize/VectorCombine.cpp b/llvm/lib/Transforms/Vectorize/VectorCombine.cpp
index 52cb1dbb33b86..95e1f96c71b48 100644
--- a/llvm/lib/Transforms/Vectorize/VectorCombine.cpp
+++ b/llvm/lib/Transforms/Vectorize/VectorCombine.cpp
@@ -289,8 +289,8 @@ bool VectorCombine::vectorizeLoadInsert(Instruction &I) {
assert(OffsetEltIndex < MinVecNumElts && "Address offset too big");
Mask[0] = OffsetEltIndex;
if (OffsetEltIndex)
- NewCost +=
- TTI.getShuffleCost(TTI::SK_PermuteSingleSrc, MinVecTy, Mask, CostKind);
+ NewCost += TTI.getShuffleCost(TTI::SK_PermuteSingleSrc, Ty, MinVecTy, Mask,
+ CostKind);
// We can aggressively convert to the vector form because the backend can
// invert this transform if it does not result in a performance win.
@@ -510,12 +510,12 @@ bool VectorCombine::isExtractExtractCheap(ExtractElementInst *Ext0,
PoisonMaskElem);
ShuffleMask[BestInsIndex] = BestExtIndex;
NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc,
- VecTy, ShuffleMask, CostKind, 0, nullptr,
- {ConvertToShuffle});
+ VecTy, VecTy, ShuffleMask, CostKind, 0,
+ nullptr, {ConvertToShuffle});
} else {
- NewCost +=
- TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc, VecTy,
- {}, CostKind, 0, nullptr, {ConvertToShuffle});
+ NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc,
+ VecTy, VecTy, {}, CostKind, 0, nullptr,
+ {ConvertToShuffle});
}
}
@@ -712,8 +712,8 @@ bool VectorCombine::foldInsExtFNeg(Instruction &I) {
InstructionCost NewCost =
TTI.getArithmeticInstrCost(Instruction::FNeg, VecTy, CostKind) +
- TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, VecTy, Mask,
- CostKind);
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, VecTy, VecTy,
+ Mask, CostKind);
bool NeedLenChg = SrcVecTy->getNumElements() != NumElts;
// If the lengths of the two vectors are not equal,
@@ -723,7 +723,7 @@ bool VectorCombine::foldInsExtFNeg(Instruction &I) {
SrcMask.assign(NumElts, PoisonMaskElem);
SrcMask[Index] = Index;
NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc,
- SrcVecTy, SrcMask, CostKind);
+ VecTy, SrcVecTy, SrcMask, CostKind);
}
if (NewCost > OldCost)
@@ -871,12 +871,12 @@ bool VectorCombine::foldBitcastShuffle(Instruction &I) {
: TargetTransformInfo::SK_PermuteTwoSrc;
InstructionCost NewCost =
- TTI.getShuffleCost(SK, NewShuffleTy, NewMask, CostKind) +
+ TTI.getShuffleCost(SK, DestTy, NewShuffleTy, NewMask, CostKind) +
(NumOps * TTI.getCastInstrCost(Instruction::BitCast, NewShuffleTy, SrcTy,
TargetTransformInfo::CastContextHint::None,
CostKind));
InstructionCost OldCost =
- TTI.getShuffleCost(SK, SrcTy, Mask, CostKind) +
+ TTI.getShuffleCost(SK, OldShuffleTy, SrcTy, Mask, CostKind) +
TTI.getCastInstrCost(Instruction::BitCast, DestTy, OldShuffleTy,
TargetTransformInfo::CastContextHint::None,
CostKind);
@@ -943,7 +943,7 @@ bool VectorCombine::scalarizeVPIntrinsic(Instruction &I) {
Mask.resize(FVTy->getNumElements(), 0);
InstructionCost SplatCost =
TTI.getVectorInstrCost(Instruction::InsertElement, VecTy, CostKind, 0) +
- TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VecTy, Mask,
+ TTI.getShuffleCost(TargetTransformInfo::SK_Broadcast, VecTy, VecTy, Mask,
CostKind);
// Calculate the cost of the VP Intrinsic
@@ -1260,14 +1260,13 @@ bool VectorCombine::foldExtractedCmps(Instruction &I) {
// ext (binop vNi1 vcmp, (shuffle vcmp, Index1)), Index0
int CheapIndex = ConvertToShuf == Ext0 ? Index1 : Index0;
int ExpensiveIndex = ConvertToShuf == Ext0 ? Index0 : Index1;
- auto *CmpTy = cast<FixedVectorType>(CmpInst::makeCmpResultType(X->getType()));
+ auto *CmpTy = cast<FixedVectorType>(CmpInst::makeCmpResultType(VecTy));
InstructionCost NewCost = TTI.getCmpSelInstrCost(
- CmpOpcode, X->getType(), CmpInst::makeCmpResultType(X->getType()), Pred,
- CostKind);
+ CmpOpcode, VecTy, CmpInst::makeCmpResultType(VecTy), Pred, CostKind);
SmallVector<int, 32> ShufMask(VecTy->getNumElements(), PoisonMaskElem);
ShufMask[CheapIndex] = ExpensiveIndex;
NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc, CmpTy,
- ShufMask, CostKind);
+ CmpTy, ShufMask, CostKind);
NewCost += TTI.getArithmeticInstrCost(I.getOpcode(), CmpTy, CostKind);
NewCost += TTI.getVectorInstrCost(*Ext0, CmpTy, CostKind, CheapIndex);
NewCost += Ext0->hasOneUse() ? 0 : Ext0Cost;
@@ -1783,8 +1782,8 @@ bool VectorCombine::foldConcatOfBoolMasks(Instruction &I) {
TTI::CastContextHint::None, CostKind);
InstructionCost NewCost = 0;
- NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, MaskTy,
- ConcatMask, CostKind);
+ NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, ConcatTy,
+ MaskTy, ConcatMask, CostKind);
NewCost += TTI.getCastInstrCost(Instruction::BitCast, ConcatIntTy, ConcatTy,
TTI::CastContextHint::None, CostKind);
if (Ty != ConcatIntTy)
@@ -1889,26 +1888,28 @@ bool VectorCombine::foldPermuteOfBinops(Instruction &I) {
// Try to merge shuffles across the binop if the new shuffles are not costly.
InstructionCost OldCost =
TTI.getArithmeticInstrCost(Opcode, BinOpTy, CostKind) +
- TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc, BinOpTy,
- OuterMask, CostKind, 0, nullptr, {BinOp}, &I);
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc, ShuffleDstTy,
+ BinOpTy, OuterMask, CostKind, 0, nullptr, {BinOp}, &I);
if (Match0)
- OldCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, Op0Ty,
- Mask0, CostKind, 0, nullptr, {Op00, Op01},
- cast<Instruction>(BinOp->getOperand(0)));
+ OldCost += TTI.getShuffleCost(
+ TargetTransformInfo::SK_PermuteTwoSrc, BinOpTy, Op0Ty, Mask0, CostKind,
+ 0, nullptr, {Op00, Op01}, cast<Instruction>(BinOp->getOperand(0)));
if (Match1)
- OldCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, Op1Ty,
- Mask1, CostKind, 0, nullptr, {Op10, Op11},
- cast<Instruction>(BinOp->getOperand(1)));
+ OldCost += TTI.getShuffleCost(
+ TargetTransformInfo::SK_PermuteTwoSrc, BinOpTy, Op1Ty, Mask1, CostKind,
+ 0, nullptr, {Op10, Op11}, cast<Instruction>(BinOp->getOperand(1)));
InstructionCost NewCost =
TTI.getArithmeticInstrCost(Opcode, ShuffleDstTy, CostKind);
if (!IsIdentity0)
- NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, Op0Ty,
- NewMask0, CostKind, 0, nullptr, {Op00, Op01});
+ NewCost +=
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, ShuffleDstTy,
+ Op0Ty, NewMask0, CostKind, 0, nullptr, {Op00, Op01});
if (!IsIdentity1)
- NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, Op1Ty,
- NewMask1, CostKind, 0, nullptr, {Op10, Op11});
+ NewCost +=
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, ShuffleDstTy,
+ Op1Ty, NewMask1, CostKind, 0, nullptr, {Op10, Op11});
LLVM_DEBUG(dbgs() << "Found a shuffle feeding a shuffled binop: " << I
<< "\n OldCost: " << OldCost << " vs NewCost: " << NewCost
@@ -2002,8 +2003,9 @@ bool VectorCombine::foldShuffleOfBinops(Instruction &I) {
InstructionCost OldCost =
TTI.getInstructionCost(LHS, CostKind) +
TTI.getInstructionCost(RHS, CostKind) +
- TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, BinResTy,
- OldMask, CostKind, 0, nullptr, {LHS, RHS}, &I);
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, ShuffleDstTy,
+ BinResTy, OldMask, CostKind, 0, nullptr, {LHS, RHS},
+ &I);
// Handle shuffle(binop(shuffle(x),y),binop(z,shuffle(w))) style patterns
// where one use shuffles have gotten split across the binop/cmp. These
@@ -2035,16 +2037,18 @@ bool VectorCombine::foldShuffleOfBinops(Instruction &I) {
ReducedInstCount |= MergeInner(Z, NumSrcElts, NewMask0, CostKind);
ReducedInstCount |= MergeInner(W, NumSrcElts, NewMask1, CostKind);
+ auto *ShuffleCmpTy =
+ FixedVectorType::get(BinOpTy->getElementType(), ShuffleDstTy);
InstructionCost NewCost =
- TTI.getShuffleCost(SK0, BinOpTy, NewMask0, CostKind, 0, nullptr, {X, Z}) +
- TTI.getShuffleCost(SK1, BinOpTy, NewMask1, CostKind, 0, nullptr, {Y, W});
+ TTI.getShuffleCost(SK0, ShuffleCmpTy, BinOpTy, NewMask0, CostKind, 0,
+ nullptr, {X, Z}) +
+ TTI.getShuffleCost(SK1, ShuffleCmpTy, BinOpTy, NewMask1, CostKind, 0,
+ nullptr, {Y, W});
if (PredLHS == CmpInst::BAD_ICMP_PREDICATE) {
NewCost +=
TTI.getArithmeticInstrCost(LHS->getOpcode(), ShuffleDstTy, CostKind);
} else {
- auto *ShuffleCmpTy =
- FixedVectorType::get(BinOpTy->getElementType(), ShuffleDstTy);
NewCost += TTI.getCmpSelInstrCost(LHS->getOpcode(), ShuffleCmpTy,
ShuffleDstTy, PredLHS, CostKind);
}
@@ -2112,15 +2116,17 @@ bool VectorCombine::foldShuffleOfSelects(Instruction &I) {
SelOp, SrcVecTy, C1VecTy, CmpInst::BAD_ICMP_PREDICATE, CostKind);
OldCost += TTI.getCmpSelInstrCost(SelOp, SrcVecTy, C2VecTy,
CmpInst::BAD_ICMP_PREDICATE, CostKind);
- OldCost += TTI.getShuffleCost(SK, SrcVecTy, Mask, CostKind, 0, nullptr,
- {I.getOperand(0), I.getOperand(1)}, &I);
+ OldCost +=
+ TTI.getShuffleCost(SK, DstVecTy, SrcVecTy, Mask, CostKind, 0, nullptr,
+ {I.getOperand(0), I.getOperand(1)}, &I);
- InstructionCost NewCost =
- TTI.getShuffleCost(SK, C1VecTy, Mask, CostKind, 0, nullptr, {C1, C2});
- NewCost +=
- TTI.getShuffleCost(SK, SrcVecTy, Mask, CostKind, 0, nullptr, {T1, T2});
- NewCost +=
- TTI.getShuffleCost(SK, SrcVecTy, Mask, CostKind, 0, nullptr, {F1, F2});
+ InstructionCost NewCost = TTI.getShuffleCost(
+ SK, FixedVectorType::get(C1VecTy->getScalarType(), Mask.size()), C1VecTy,
+ Mask, CostKind, 0, nullptr, {C1, C2});
+ NewCost += TTI.getShuffleCost(SK, DstVecTy, SrcVecTy, Mask, CostKind, 0,
+ nullptr, {T1, T2});
+ NewCost += TTI.getShuffleCost(SK, DstVecTy, SrcVecTy, Mask, CostKind, 0,
+ nullptr, {F1, F2});
auto *C1C2ShuffledVecTy = cast<FixedVectorType>(
toVectorTy(Type::getInt1Ty(I.getContext()), DstVecTy->getNumElements()));
NewCost += TTI.getCmpSelInstrCost(SelOp, DstVecTy, C1C2ShuffledVecTy,
@@ -2220,11 +2226,12 @@ bool VectorCombine::foldShuffleOfCastops(Instruction &I) {
TTI::CastContextHint::None, CostKind);
InstructionCost OldCost = CostC0 + CostC1;
OldCost +=
- TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, CastDstTy,
- OldMask, CostKind, 0, nullptr, {}, &I);
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, ShuffleDstTy,
+ CastDstTy, OldMask, CostKind, 0, nullptr, {}, &I);
- InstructionCost NewCost = TTI.getShuffleCost(
- TargetTransformInfo::SK_PermuteTwoSrc, CastSrcTy, NewMask, CostKind);
+ InstructionCost NewCost =
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, NewShuffleDstTy,
+ CastSrcTy, NewMask, CostKind);
NewCost += TTI.getCastInstrCost(Opcode, ShuffleDstTy, NewShuffleDstTy,
TTI::CastContextHint::None, CostKind);
if (!C0->hasOneUse())
@@ -2363,8 +2370,9 @@ bool VectorCombine::foldShuffleOfShuffles(Instruction &I) {
TargetTransformInfo::ShuffleKind SK =
IsUnary ? TargetTransformInfo::SK_PermuteSingleSrc
: TargetTransformInfo::SK_PermuteTwoSrc;
- InstructionCost NewCost = TTI.getShuffleCost(
- SK, ShuffleSrcTy, NewMask, CostKind, 0, nullptr, {NewX, NewY});
+ InstructionCost NewCost =
+ TTI.getShuffleCost(SK, ShuffleDstTy, ShuffleSrcTy, NewMask, CostKind, 0,
+ nullptr, {NewX, NewY});
if (!OuterV0->hasOneUse())
NewCost += InnerCost0;
if (!OuterV1->hasOneUse())
@@ -2415,21 +2423,23 @@ bool VectorCombine::foldShuffleOfIntrinsics(Instruction &I) {
InstructionCost OldCost =
TTI.getIntrinsicInstrCost(IntrinsicCostAttributes(IID, *II0), CostKind) +
TTI.getIntrinsicInstrCost(IntrinsicCostAttributes(IID, *II1), CostKind) +
- TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, II0Ty, OldMask,
- CostKind, 0, nullptr, {II0, II1}, &I);
+ TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc, ShuffleDstTy,
+ II0Ty, OldMask, CostKind, 0, nullptr, {II0, II1}, &I);
SmallVector<Type *> NewArgsTy;
InstructionCost NewCost = 0;
- for (unsigned I = 0, E = II0->arg_size(); I != E; ++I)
+ for (unsigned I = 0, E = II0->arg_size(); I != E; ++I) {
if (isVectorIntrinsicWithScalarOpAtArg(IID, I, &TTI)) {
NewArgsTy.push_back(II0->getArgOperand(I)->getType());
} else {
auto *VecTy = cast<FixedVectorType>(II0->getArgOperand(I)->getType());
- NewArgsTy.push_back(FixedVectorType::get(VecTy->getElementType(),
- ShuffleDstTy->getNumElements()));
+ auto *ArgTy = FixedVectorType::get(VecTy->getElementType(),
+ ShuffleDstTy->getNumElements());
+ NewArgsTy.push_back(ArgTy);
NewCost += TTI.getShuffleCost(TargetTransformInfo::SK_PermuteTwoSrc,
- VecTy, OldMask, CostKind);
+ ArgTy, VecTy, OldMask, CostKind);
}
+ }
IntrinsicCostAttributes NewAttr(IID, ShuffleDstTy, NewArgsTy);
NewCost += TTI.getIntrinsicInstrCost(NewAttr, CostKind);
@@ -2508,7 +2518,9 @@ static bool isFreeConcat(ArrayRef<InstLane> Item, TTI::TargetCostKind CostKind,
// during legalization.
SmallVector<int, 16> ConcatMask(NumElts * 2);
std::iota(ConcatMask.begin(), ConcatMask.end(), 0);
- if (TTI.getShuffleCost(TTI::SK_PermuteTwoSrc, Ty, ConcatMask, CostKind) != 0)
+ if (TTI.getShuffleCost(TTI::SK_PermuteTwoSrc,
+ FixedVectorType::get(Ty->getScalarType(), NumElts * 2),
+ Ty, ConcatMask, CostKind) != 0)
return false;
unsigned NumSlices = Item.size() / NumElts;
@@ -2877,21 +2889,15 @@ bool VectorCombine::foldShuffleFromReductions(Instruction &I) {
SmallVector<int> ConcatMask;
Shuffle->getShuffleMask(ConcatMask);
sort(ConcatMask, [](int X, int Y) { return (unsigned)X < (unsigned)Y; });
- // In the case of a truncating shuffle it's possible for the mask
- // to have an index greater than the size of the resulting vector.
- // This requires special handling.
- bool IsTruncatingShuffle = VecType->getNumElements() < NumInputElts;
bool UsesSecondVec =
any_of(ConcatMask, [&](int M) { return M >= (int)NumInputElts; });
- FixedVectorType *VecTyForCost =
- (UsesSecondVec && !IsTruncatingShuffle) ? VecType : ShuffleInputType;
InstructionCost OldCost = TTI.getShuffleCost(
- UsesSecondVec ? TTI::SK_PermuteTwoSrc : TTI::SK_PermuteSingleSrc,
- VecTyForCost, Shuffle->getShuffleMask(), CostKind);
+ UsesSecondVec ? TTI::SK_PermuteTwoSrc : TTI::SK_PermuteSingleSrc, VecType,
+ ShuffleInputType, Shuffle->getShuffleMask(), CostKind);
InstructionCost NewCost = TTI.getShuffleCost(
- UsesSecondVec ? TTI::SK_PermuteTwoSrc : TTI::SK_PermuteSingleSrc,
- VecTyForCost, ConcatMask, CostKind);
+ UsesSecondVec ? TTI::SK_PermuteTwoSrc : TTI::SK_PermuteSingleSrc, VecType,
+ ShuffleInputType, ConcatMask, CostKind);
LLVM_DEBUG(dbgs() << "Found a reduction feeding from a shuffle: " << *Shuffle
<< "\n");
@@ -3205,10 +3211,11 @@ bool VectorCombine::foldSelectShuffle(Instruction &I, bool FromReduction) {
return C + TTI.getShuffleCost(isa<UndefValue>(SV->getOperand(1))
? TTI::SK_PermuteSingleSrc
: TTI::SK_PermuteTwoSrc,
- VT, SV->getShuffleMask(), CostKind);
+ VT, VT, SV->getShuffleMask(), CostKind);
};
auto AddShuffleMaskCost = [&](InstructionCost C, ArrayRef<int> Mask) {
- return C + TTI.getShuffleCost(TTI::SK_PermuteTwoSrc, VT, Mask, CostKind);
+ return C +
+ TTI.getShuffleCost(TTI::SK_PermuteTwoSrc, VT, VT, Mask, CostKind);
};
// Get the costs of the shuffles + binops before and after with the new
@@ -3446,8 +3453,8 @@ bool VectorCombine::foldInsExtVectorToShuffle(Instruction &I) {
// Ignore 'free' identity insertion shuffle.
// TODO: getShuffleCost should return TCC_Free for Identity shuffles.
if (!ShuffleVectorInst::isIdentityMask(Mask, NumSrcElts))
- NewCost += TTI.getShuffleCost(SK, DstVecTy, Mask, CostKind, 0, nullptr,
- {DstVec, SrcVec});
+ NewCost += TTI.getShuffleCost(SK, DstVecTy, DstVecTy, Mask, CostKind, 0,
+ nullptr, {DstVec, SrcVec});
} else {
// When creating length-changing-vector, always create with a Mask whose
// first element has an ExtIdx, so that the first element of the vector
@@ -3459,8 +3466,8 @@ bool VectorCombine::foldInsExtVectorToShuffle(Instruction &I) {
ExtToVecMask[0] = ExtIdx;
// Add cost for expanding or narrowing
NewCost = TTI.getShuffleCost(TargetTransformInfo::SK_PermuteSingleSrc,
- SrcVecTy, ExtToVecMask, CostKind);
- NewCost += TTI.getShuffleCost(SK, DstVecTy, Mask, CostKind);
+ DstVecTy, SrcVecTy, ExtToVecMask, CostKind);
+ NewCost += TTI.getShuffleCost(SK, DstVecTy, DstVecTy, Mask, CostKind);
}
if (!Ext->hasOneUse())
>From 3735b0b4c706cd89b69683f5dd85b7121aec05de Mon Sep 17 00:00:00 2001
From: David Green <david.green at arm.com>
Date: Fri, 30 May 2025 15:28:18 +0100
Subject: [PATCH 2/2] Rebase and update comments
---
.../llvm/Analysis/TargetTransformInfo.h | 24 +++++++++----------
.../AArch64/AArch64TargetTransformInfo.cpp | 3 ++-
2 files changed, 13 insertions(+), 14 deletions(-)
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index 6335efc243813..22b401fc0c74e 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -1383,19 +1383,17 @@ class TargetTransformInfo {
/// \return The cost of a shuffle instruction of kind Kind with inputs of type
/// SrcTy, producing a vector of type DstTy. The exact mask may be passed as
- /// Mask, or else the array will be empty. The index and subtype parameters
- /// are used by the subvector insertion and extraction shuffle kinds to show
- /// the insert/extract point and the type of the subvector being
- /// inserted/extracted. The operands of the shuffle can be passed through \p
- /// Args, which helps improve the cost estimation in some cases, like in
- /// broadcast loads.
- LLVM_ABI InstructionCost
- getShuffleCost(ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
- ArrayRef<int> Mask = {},
- TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput,
- int Index = 0, VectorType *SubTp = nullptr,
- ArrayRef<const Value *> Args = {},
- const Instruction *CxtI = nullptr) const;
+ /// Mask, or else the array will be empty. The Index and SubTp parameters
+ /// are used by the subvector insertions shuffle kinds to show the insert
+ /// point and the type of the subvector being inserted. The operands of the
+ /// shuffle can be passed through \p Args, which helps improve the cost
+ /// estimation in some cases, like in broadcast loads.
+ LLVM_ABI InstructionCost getShuffleCost(
+ ShuffleKind Kind, VectorType *DstTy, VectorType *SrcTy,
+ ArrayRef<int> Mask = {},
+ TTI::TargetCostKind CostKind = TTI::TCK_RecipThroughput, int Index = 0,
+ VectorType *SubTp = nullptr, ArrayRef<const Value *> Args = {},
+ const Instruction *CxtI = nullptr) const;
/// Represents a hint about the context in which a cast is used.
///
diff --git a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
index ad582d45e77da..6df1d0bd437e9 100644
--- a/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
+++ b/llvm/lib/Target/AArch64/AArch64TargetTransformInfo.cpp
@@ -5538,7 +5538,8 @@ AArch64TTIImpl::getShuffleCost(TTI::ShuffleKind Kind, VectorType *DstTy,
NumSources <= 2
? getShuffleCost(NumSources <= 1 ? TTI::SK_PermuteSingleSrc
: TTI::SK_PermuteTwoSrc,
- NTp, NTp, NMask, CostKind, 0, nullptr, Args, CxtI)
+ NTp, NTp, NMask, CostKind, 0, nullptr, Args,
+ CxtI)
: LTNumElts;
Result.first->second = NCost;
Cost += NCost;
More information about the llvm-commits
mailing list