[llvm] a68dcd0 - [TTI] Use users of GEP to guess access type in getGEPCost
Luke Lau via llvm-commits
llvm-commits at lists.llvm.org
Thu Jun 29 05:45:03 PDT 2023
Author: Luke Lau
Date: 2023-06-29T13:44:37+01:00
New Revision: a68dcd09e8084dc179fb3218e8953417c4b7474a
URL: https://github.com/llvm/llvm-project/commit/a68dcd09e8084dc179fb3218e8953417c4b7474a
DIFF: https://github.com/llvm/llvm-project/commit/a68dcd09e8084dc179fb3218e8953417c4b7474a.diff
LOG: [TTI] Use users of GEP to guess access type in getGEPCost
Currently getGEPCost uses the target type of the GEP as a heuristic for
the type that will be accessed, to pass onto isLegalAddressingMode.
Targets use this to work out if a GEP can then be folded into the
load/store instruction that uses the GEP.
For example, on RISC-V loads and stores can have an offset added to a
base register folded into a single instruction, so the following GEP is
free:
%p = getelementptr i32, ptr %base, i32 42 ; getInstructionCost = 0
%x = load i32, ptr %p ; getInstructionCost = 1
------------------------------------------------------------------------
lw t0, a0(42)
However vector loads and stores cannot have an offset folded into them,
so the following GEP is costed:
%p = getelementptr <2 x i32>, ptr %base, i32 42 ; getInstructionCost = 1
%x = load <2 x i32>, ptr %p ; getInstructionCost = 1
------------------------------------------------------------------------
addi a0, 42
vle32 v8, (a0)
The issue arises whenever there is a mismatch between the target type of
the GEP and the type that is actually accessed:
%p = getelementptr i32, ptr %base, i32 42 ; getInstructionCost = 0
%x = load <2 x i32>, ptr %p ; getInstructionCost = 1
------------------------------------------------------------------------
addi a0, 42
vle32 v8, (a0)
Even though this GEP will result in an add instruction, because TTI
thinks it's loading an i32, it will think it can be folded and not
charge for it.
The target type can become mismatched with the memory access during
transformations, noticeably during SLP where a scalar base pointer will
be reused to perform a vector load or store.
This patch adds an optional AccessType argument to getGEPCost which
allows the type of memory accessed by users to be passed in as a hint,
so that we can more accurately determine if the GEP can be folded into
its users.
If AccessType is not provided, getGEPCost falls back to the old
behaviour of using the PointeeType to guess the memory access type. This
can be revisited in a later patch.
Also for now, only GEPs with exactly one user use the access type hint.
Whilst we could look through all users and use all access types to
determine if we can fold the GEP, this patch avoids doing so to prevent
O(N) behaviour.
Differential Revision: https://reviews.llvm.org/D149889
Added:
Modified:
llvm/include/llvm/Analysis/TargetTransformInfo.h
llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
llvm/include/llvm/CodeGen/BasicTTIImpl.h
llvm/lib/Analysis/TargetTransformInfo.cpp
llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
llvm/lib/Target/X86/X86TargetTransformInfo.cpp
llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
llvm/test/Analysis/CostModel/ARM/mve-gather-scatter-cost.ll
llvm/test/Analysis/CostModel/RISCV/gep.ll
Removed:
################################################################################
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfo.h b/llvm/include/llvm/Analysis/TargetTransformInfo.h
index 0b679039ec4ffa..ca4b83727c02c4 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfo.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfo.h
@@ -284,9 +284,19 @@ class TargetTransformInfo {
};
/// Estimate the cost of a GEP operation when lowered.
+ ///
+ /// \p PointeeType is the source element type of the GEP.
+ /// \p Ptr is the base pointer operand.
+ /// \p Operands is the list of indices following the base pointer.
+ ///
+ /// \p AccessType is a hint as to what type of memory might be accessed by
+ /// users of the GEP. getGEPCost will use it to determine if the GEP can be
+ /// folded into the addressing mode of a load/store. If AccessType is null,
+ /// then the resulting target type based off of PointeeType will be used as an
+ /// approximation.
InstructionCost
getGEPCost(Type *PointeeType, const Value *Ptr,
- ArrayRef<const Value *> Operands,
+ ArrayRef<const Value *> Operands, Type *AccessType = nullptr,
TargetCostKind CostKind = TCK_SizeAndLatency) const;
/// Describe known properties for a set of pointers.
@@ -1680,6 +1690,7 @@ class TargetTransformInfo::Concept {
virtual const DataLayout &getDataLayout() const = 0;
virtual InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
ArrayRef<const Value *> Operands,
+ Type *AccessType,
TTI::TargetCostKind CostKind) = 0;
virtual InstructionCost
getPointersChainCost(ArrayRef<const Value *> Ptrs, const Value *Base,
@@ -2041,9 +2052,9 @@ class TargetTransformInfo::Model final : public TargetTransformInfo::Concept {
InstructionCost
getGEPCost(Type *PointeeType, const Value *Ptr,
- ArrayRef<const Value *> Operands,
+ ArrayRef<const Value *> Operands, Type *AccessType,
TargetTransformInfo::TargetCostKind CostKind) override {
- return Impl.getGEPCost(PointeeType, Ptr, Operands, CostKind);
+ return Impl.getGEPCost(PointeeType, Ptr, Operands, AccessType, CostKind);
}
InstructionCost getPointersChainCost(ArrayRef<const Value *> Ptrs,
const Value *Base,
diff --git a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
index a78d187d61a437..f9f24b3faa2605 100644
--- a/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
+++ b/llvm/include/llvm/Analysis/TargetTransformInfoImpl.h
@@ -47,7 +47,7 @@ class TargetTransformInfoImplBase {
const DataLayout &getDataLayout() const { return DL; }
InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
- ArrayRef<const Value *> Operands,
+ ArrayRef<const Value *> Operands, Type *AccessType,
TTI::TargetCostKind CostKind) const {
// In the basic model, we just assume that all-constant GEPs will be folded
// into their uses via addressing modes.
@@ -987,7 +987,7 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
using BaseT::getGEPCost;
InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
- ArrayRef<const Value *> Operands,
+ ArrayRef<const Value *> Operands, Type *AccessType,
TTI::TargetCostKind CostKind) {
assert(PointeeType && Ptr && "can't get GEPCost of nullptr");
assert(cast<PointerType>(Ptr->getType()->getScalarType())
@@ -1041,11 +1041,29 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
}
}
+ // If we haven't been provided a hint, use the target type for now.
+ //
+ // TODO: Take a look at potentially removing this: This is *slightly* wrong
+ // as it's possible to have a GEP with a foldable target type but a memory
+ // access that isn't foldable. For example, this load isn't foldable on
+ // RISC-V:
+ //
+ // %p = getelementptr i32, ptr %base, i32 42
+ // %x = load <2 x i32>, ptr %p
+ if (!AccessType)
+ AccessType = TargetType;
+
+ // If the final address of the GEP is a legal addressing mode for the given
+ // access type, then we can fold it into its users.
if (static_cast<T *>(this)->isLegalAddressingMode(
- TargetType, const_cast<GlobalValue *>(BaseGV),
+ AccessType, const_cast<GlobalValue *>(BaseGV),
BaseOffset.sextOrTrunc(64).getSExtValue(), HasBaseReg, Scale,
Ptr->getType()->getPointerAddressSpace()))
return TTI::TCC_Free;
+
+ // TODO: Instead of returning TCC_Basic here, we should use
+ // getArithmeticInstrCost. Or better yet, provide a hook to let the target
+ // model it.
return TTI::TCC_Basic;
}
@@ -1080,7 +1098,7 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
SmallVector<const Value *> Indices(GEP->indices());
Cost += static_cast<T *>(this)->getGEPCost(GEP->getSourceElementType(),
GEP->getPointerOperand(),
- Indices, CostKind);
+ Indices, nullptr, CostKind);
}
}
return Cost;
@@ -1132,9 +1150,15 @@ class TargetTransformInfoImplCRTPBase : public TargetTransformInfoImplBase {
break;
case Instruction::GetElementPtr: {
const auto *GEP = cast<GEPOperator>(U);
+ Type *AccessType = nullptr;
+ // For now, only provide the AccessType in the simple case where the GEP
+ // only has one user.
+ if (GEP->hasOneUser() && I)
+ AccessType = I->user_back()->getAccessType();
+
return TargetTTI->getGEPCost(GEP->getSourceElementType(),
Operands.front(), Operands.drop_front(),
- CostKind);
+ AccessType, CostKind);
}
case Instruction::Add:
case Instruction::FAdd:
diff --git a/llvm/include/llvm/CodeGen/BasicTTIImpl.h b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
index ca7f6d34b66bd1..2dc81596573ef0 100644
--- a/llvm/include/llvm/CodeGen/BasicTTIImpl.h
+++ b/llvm/include/llvm/CodeGen/BasicTTIImpl.h
@@ -420,9 +420,9 @@ class BasicTTIImplBase : public TargetTransformInfoImplCRTPBase<T> {
}
InstructionCost getGEPCost(Type *PointeeType, const Value *Ptr,
- ArrayRef<const Value *> Operands,
+ ArrayRef<const Value *> Operands, Type *AccessType,
TTI::TargetCostKind CostKind) {
- return BaseT::getGEPCost(PointeeType, Ptr, Operands, CostKind);
+ return BaseT::getGEPCost(PointeeType, Ptr, Operands, AccessType, CostKind);
}
unsigned getEstimatedNumberOfCaseClusters(const SwitchInst &SI,
diff --git a/llvm/lib/Analysis/TargetTransformInfo.cpp b/llvm/lib/Analysis/TargetTransformInfo.cpp
index 435baab3f4424c..e5a8d7934a0017 100644
--- a/llvm/lib/Analysis/TargetTransformInfo.cpp
+++ b/llvm/lib/Analysis/TargetTransformInfo.cpp
@@ -226,11 +226,10 @@ int TargetTransformInfo::getInlinerVectorBonusPercent() const {
return TTIImpl->getInlinerVectorBonusPercent();
}
-InstructionCost
-TargetTransformInfo::getGEPCost(Type *PointeeType, const Value *Ptr,
- ArrayRef<const Value *> Operands,
- TTI::TargetCostKind CostKind) const {
- return TTIImpl->getGEPCost(PointeeType, Ptr, Operands, CostKind);
+InstructionCost TargetTransformInfo::getGEPCost(
+ Type *PointeeType, const Value *Ptr, ArrayRef<const Value *> Operands,
+ Type *AccessType, TTI::TargetCostKind CostKind) const {
+ return TTIImpl->getGEPCost(PointeeType, Ptr, Operands, AccessType, CostKind);
}
InstructionCost TargetTransformInfo::getPointersChainCost(
diff --git a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
index 2580789183b564..328ff8806eaa7e 100644
--- a/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
+++ b/llvm/lib/Target/RISCV/RISCVTargetTransformInfo.cpp
@@ -1641,7 +1641,7 @@ InstructionCost RISCVTTIImpl::getPointersChainCost(
} else {
SmallVector<const Value *> Indices(GEP->indices());
Cost += getGEPCost(GEP->getSourceElementType(), GEP->getPointerOperand(),
- Indices, CostKind);
+ Indices, nullptr, CostKind);
}
}
return Cost;
diff --git a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
index 7a5baa82fda228..4c678a6ff846dc 100644
--- a/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
+++ b/llvm/lib/Target/X86/X86TargetTransformInfo.cpp
@@ -4969,7 +4969,8 @@ X86TTIImpl::getPointersChainCost(ArrayRef<const Value *> Ptrs,
if (const auto *BaseGEP = dyn_cast<GetElementPtrInst>(Base)) {
SmallVector<const Value *> Indices(BaseGEP->indices());
return getGEPCost(BaseGEP->getSourceElementType(),
- BaseGEP->getPointerOperand(), Indices, CostKind);
+ BaseGEP->getPointerOperand(), Indices, nullptr,
+ CostKind);
}
return TTI::TCC_Free;
}
diff --git a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
index 77a7852af9828e..a9a5cc55bc3f2c 100644
--- a/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
+++ b/llvm/lib/Transforms/Vectorize/SLPVectorizer.cpp
@@ -7454,7 +7454,8 @@ BoUpSLP::getEntryCost(const TreeEntry *E, ArrayRef<Value *> VectorizedVals,
if (const auto *Base = dyn_cast<GetElementPtrInst>(BasePtr)) {
SmallVector<const Value *> Indices(Base->indices());
VecCost = TTI->getGEPCost(Base->getSourceElementType(),
- Base->getPointerOperand(), Indices, CostKind);
+ Base->getPointerOperand(), Indices, nullptr,
+ CostKind);
}
}
diff --git a/llvm/test/Analysis/CostModel/ARM/mve-gather-scatter-cost.ll b/llvm/test/Analysis/CostModel/ARM/mve-gather-scatter-cost.ll
index fd6b044680fa39..fa18f4724c3984 100644
--- a/llvm/test/Analysis/CostModel/ARM/mve-gather-scatter-cost.ll
+++ b/llvm/test/Analysis/CostModel/ARM/mve-gather-scatter-cost.ll
@@ -526,7 +526,7 @@ define void @gep_v16i8(ptr %base, ptr %base16, <16 x i8> %ind8, <16 x i32> %ind3
; CHECK-NEXT: Cost Model: Found an estimated cost of 224 for instruction: %resbs = call <16 x i8> @llvm.masked.gather.v16i8.v16p0(<16 x ptr> %gepbsb, i32 2, <16 x i1> %mask, <16 x i8> undef)
; CHECK-NEXT: Cost Model: Found an estimated cost of 224 for instruction: call void @llvm.masked.scatter.v16i8.v16p0(<16 x i8> %resbs, <16 x ptr> %gepbsb, i32 2, <16 x i1> %mask)
; CHECK-NEXT: Cost Model: Found an estimated cost of 74 for instruction: %indzext4 = zext <16 x i8> %ind8 to <16 x i32>
-; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %gep4 = getelementptr i8, ptr %base, <16 x i32> %indzext
+; CHECK-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %gep4 = getelementptr i8, ptr %base, <16 x i32> %indzext
; CHECK-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %indtrunc = trunc <16 x i32> %ind32 to <16 x i8>
; CHECK-NEXT: Cost Model: Found an estimated cost of 32 for instruction: call void @llvm.masked.scatter.v16i8.v16p0(<16 x i8> %indtrunc, <16 x ptr> %gep4, i32 2, <16 x i1> %mask)
; CHECK-NEXT: Cost Model: Found an estimated cost of 0 for instruction: ret void
diff --git a/llvm/test/Analysis/CostModel/RISCV/gep.ll b/llvm/test/Analysis/CostModel/RISCV/gep.ll
index 4f1c4db022b5d2..c7a3e5d30aba7f 100644
--- a/llvm/test/Analysis/CostModel/RISCV/gep.ll
+++ b/llvm/test/Analysis/CostModel/RISCV/gep.ll
@@ -261,29 +261,29 @@ define void @testvecs(i32 %i) {
; be folded into the instruction.
define void @non_foldable_vector_uses(ptr %base, <2 x ptr> %base.vec) {
; RVI-LABEL: 'non_foldable_vector_uses'
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %1 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %1 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %x1 = load volatile <2 x i8>, ptr %1, align 2
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %2 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %2 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %x2 = call <2 x i8> @llvm.masked.load.v2i8.p0(ptr %2, i32 1, <2 x i1> undef, <2 x i8> undef)
; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %3 = getelementptr i8, <2 x ptr> %base.vec, <2 x i32> <i32 42, i32 43>
; RVI-NEXT: Cost Model: Found an estimated cost of 2 for instruction: %x3 = call <2 x i8> @llvm.masked.gather.v2i8.v2p0(<2 x ptr> %3, i32 1, <2 x i1> undef, <2 x i8> undef)
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %4 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %4 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 5 for instruction: %x4 = call <2 x i8> @llvm.masked.expandload.v2i8(ptr %4, <2 x i1> undef, <2 x i8> undef)
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %5 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %5 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 5 for instruction: %x5 = call <2 x i8> @llvm.vp.load.v2i8.p0(ptr %5, <2 x i1> undef, i32 undef)
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %6 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %6 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 5 for instruction: %x6 = call <2 x i8> @llvm.experimental.vp.strided.load.v2i8.p0.i64(ptr %6, i64 undef, <2 x i1> undef, i32 undef)
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %7 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %7 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: store volatile <2 x i8> undef, ptr %7, align 2
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %8 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %8 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: call void @llvm.masked.store.v2i8.p0(<2 x i8> undef, ptr %8, i32 1, <2 x i1> undef)
; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %9 = getelementptr i8, <2 x ptr> %base.vec, <2 x i32> <i32 42, i32 43>
; RVI-NEXT: Cost Model: Found an estimated cost of 2 for instruction: call void @llvm.masked.scatter.v2i8.v2p0(<2 x i8> undef, <2 x ptr> %9, i32 1, <2 x i1> undef)
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %10 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %10 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.masked.compressstore.v2i8(<2 x i8> undef, ptr %10, <2 x i1> undef)
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %11 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %11 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.vp.store.v2i8.p0(<2 x i8> undef, ptr %11, <2 x i1> undef, i32 undef)
-; RVI-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %12 = getelementptr i8, ptr %base, i32 42
+; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %12 = getelementptr i8, ptr %base, i32 42
; RVI-NEXT: Cost Model: Found an estimated cost of 12 for instruction: call void @llvm.experimental.vp.strided.store.v2i8.p0.i64(<2 x i8> undef, ptr %12, i64 undef, <2 x i1> undef, i32 undef)
; RVI-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
;
More information about the llvm-commits
mailing list