[llvm] [AMDGPU] Price the packed form of a vector of i1 (PR #217327)
via llvm-commits
llvm-commits at lists.llvm.org
Thu Aug 20 04:10:21 PDT 2026
https://github.com/michaelselehov updated https://github.com/llvm/llvm-project/pull/217327
>From 027ef7dd183df43459d98bf74681857bb03d0c21 Mon Sep 17 00:00:00 2001
From: Michael Selehov <michael.selehov at amd.com>
Date: Fri, 14 Aug 2026 09:35:39 -0500
Subject: [PATCH] [AMDGPU] Price the packed form of a vector of i1
A vector of i1 has no packed form on this target. Every element lives in its own
mask. When an instruction needs the elements as the bits of one integer, the
target has to build that integer: it selects a 0 or a 1 per element, shifts the
value into place and merges it. The generic cost model charges about one
instruction per element for that work.
I counted instructions after codegen on gfx900, gfx942, gfx1030, gfx1100 and
gfx1201, at 8, 16 and 32 elements. The counts use a divergent operand. They
exclude a baseline of the same loads and compares. They also exclude s_delay_alu
and s_wait_alu, which GFX11 and GFX12 need between dependent operations and
which no cost model counts.
To build the integer takes 4.0 to 4.8 instructions per element. To take it apart
takes 2.6 to 2.9 per element. The work does not depend on the subtarget, so
neither does the cost. This patch charges 4 and 3.
Three reductions over a vector of i1 use the packed form, and the packing
dominates their cost. On gfx1030, at 8 elements:
reduction real generic model this patch
and 34 9 32
or 33 9 32
add 33 15 32
The number for the add is the form that the SLP vectorizer emits, a bit count
over the packed mask. A xor reduction is a parity computation. It stays in the
masks, and the generic model already prices it well, so this patch leaves it
alone.
The generic model also overprices a mul reduction and a min or a max reduction
over a vector of i1. It asks 36 and 22 against 4 real instructions, because
those reductions stay in 32-bit lanes. A change there can create new
vectorization, so it needs its own measurements. I leave it for a separate
patch.
One case now gets a cost that is too high. The add reduction intrinsic over a
vector of i1 lowers to a parity chain of about 2 instructions per element, and
this patch charges 4. The cost model does not see which form the caller will
build. The SLP vectorizer builds the packed form whenever the result is wider
than i1.
The add cost is the one that fixes a regression in the rocSPARSE csrgemm kernel,
where SLP replaced a cheap scalar chain of adds with the packed form. The and,
the or and the bitcast costs change no code in that module: its assembly is the
same with and without them.
Assisted-by: Claude Opus
---
.../AMDGPU/AMDGPUTargetTransformInfo.cpp | 56 +++++++++
.../Target/AMDGPU/AMDGPUTargetTransformInfo.h | 5 +
.../Analysis/CostModel/AMDGPU/bitcast-mask.ll | 82 +++++++++++++
.../CostModel/AMDGPU/reduce-add-i1.ll | 57 +++++++++
.../Analysis/CostModel/AMDGPU/reduce-and.ll | 32 ++---
.../Analysis/CostModel/AMDGPU/reduce-or.ll | 32 ++---
.../AMDGPU/reduction-i1-ctpop-cost.ll | 109 ++++++++++++++++++
7 files changed, 341 insertions(+), 32 deletions(-)
create mode 100644 llvm/test/Analysis/CostModel/AMDGPU/bitcast-mask.ll
create mode 100644 llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll
create mode 100644 llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
index 980e26082064f..36f09dc55bc16 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
@@ -980,6 +980,48 @@ InstructionCost GCNTTIImpl::getCFInstrCost(unsigned Opcode,
return BaseT::getCFInstrCost(Opcode, CostKind, I);
}
+// A vector of i1 has no packed form on this target. Every element lives in its
+// own mask. An instruction that needs the elements as the bits of an integer
+// has to build that integer, and an instruction that starts from an integer has
+// to take it apart. To build it, the target selects a 0 or a 1 per element,
+// shifts the value into place and merges it. To take it apart, it shifts, masks
+// and compares per element. Instruction counts after codegen give 4.0 to 4.8
+// per element to build and 2.6 to 2.9 per element to take apart, on every
+// generation from GFX9 to GFX12.
+static constexpr unsigned MaskPackCostPerElt = 4;
+static constexpr unsigned MaskUnpackCostPerElt = 3;
+
+/// Returns the number of elements when \p Ty is a fixed vector of i1 that needs
+/// packing, and nullopt otherwise. A single element needs no packing.
+static std::optional<unsigned> getPackedMaskElts(Type *Ty) {
+ auto *FVT = dyn_cast<FixedVectorType>(Ty);
+ if (FVT && FVT->getElementType()->isIntegerTy(1) && FVT->getNumElements() > 1)
+ return FVT->getNumElements();
+ return std::nullopt;
+}
+
+InstructionCost GCNTTIImpl::getCastInstrCost(unsigned Opcode, Type *Dst,
+ Type *Src,
+ TTI::CastContextHint CCH,
+ TTI::TargetCostKind CostKind,
+ const Instruction *I) const {
+ // A bitcast between a vector of i1 and an integer packs or unpacks a mask.
+ // The generic model charges one instruction per element to pack, and nothing
+ // at all to unpack.
+ if (Opcode == Instruction::BitCast) {
+ if (std::optional<unsigned> Elts = getPackedMaskElts(Src);
+ Elts && Dst->isIntegerTy())
+ return InstructionCost(MaskPackCostPerElt) * *Elts *
+ getFullRateInstrCost();
+ if (std::optional<unsigned> Elts = getPackedMaskElts(Dst);
+ Elts && Src->isIntegerTy())
+ return InstructionCost(MaskUnpackCostPerElt) * *Elts *
+ getFullRateInstrCost();
+ }
+
+ return BaseT::getCastInstrCost(Opcode, Dst, Src, CCH, CostKind, I);
+}
+
InstructionCost
GCNTTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *Ty,
std::optional<FastMathFlags> FMF,
@@ -987,6 +1029,20 @@ GCNTTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *Ty,
if (TTI::requiresOrderedReduction(FMF))
return BaseT::getArithmeticReductionCost(Opcode, Ty, FMF, CostKind);
+ // An add reduction, an and reduction and an or reduction over a vector of i1
+ // all go through the packed form of the mask. The add counts the bits of the
+ // packed integer. The and and the or compare that integer against all ones or
+ // against zero. The packing dominates the cost. The generic model prices the
+ // reduction as a shuffle tree, which is far too cheap. A xor reduction is a
+ // parity computation. It stays in the masks, and the generic model prices it
+ // well.
+ if (Opcode == Instruction::Add || Opcode == Instruction::And ||
+ Opcode == Instruction::Or) {
+ if (std::optional<unsigned> Elts = getPackedMaskElts(Ty))
+ return InstructionCost(MaskPackCostPerElt) * *Elts *
+ getFullRateInstrCost();
+ }
+
EVT OrigTy = TLI->getValueType(DL, Ty);
// Computes cost on targets that have packed math instructions(which support
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h
index df7b6d339e6c2..2be0a5ef329cb 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.h
@@ -268,6 +268,11 @@ class GCNTTIImpl final : public BasicTTIImplBase<GCNTTIImpl> {
return InlinerVectorBonusPercent;
}
+ InstructionCost
+ getCastInstrCost(unsigned Opcode, Type *Dst, Type *Src,
+ TTI::CastContextHint CCH, TTI::TargetCostKind CostKind,
+ const Instruction *I = nullptr) const override;
+
InstructionCost
getArithmeticReductionCost(unsigned Opcode, VectorType *Ty,
std::optional<FastMathFlags> FMF,
diff --git a/llvm/test/Analysis/CostModel/AMDGPU/bitcast-mask.ll b/llvm/test/Analysis/CostModel/AMDGPU/bitcast-mask.ll
new file mode 100644
index 0000000000000..8caa003ea181d
--- /dev/null
+++ b/llvm/test/Analysis/CostModel/AMDGPU/bitcast-mask.ll
@@ -0,0 +1,82 @@
+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -mtriple=amdgpu9.00-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu10.30-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -mattr=-real-true16 -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -passes="print<cost-model>" -cost-kind=code-size 2>&1 -disable-output | FileCheck -check-prefixes=ALL-SIZE %s
+; END.
+
+; A vector of i1 has no packed form on this target, so a bitcast to an integer
+; builds the integer out of the masks, and a bitcast from an integer takes it
+; apart. Both costs are per element, and neither depends on the subtarget.
+
+define void @pack() {
+; ALL-LABEL: 'pack'
+; ALL-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = bitcast <2 x i1> poison to i2
+; ALL-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = bitcast <4 x i1> poison to i4
+; ALL-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = bitcast <8 x i1> poison to i8
+; ALL-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = bitcast <16 x i1> poison to i16
+; ALL-NEXT: Cost Model: Found an estimated cost of 128 for instruction: %V32 = bitcast <32 x i1> poison to i32
+; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret void
+;
+; ALL-SIZE-LABEL: 'pack'
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = bitcast <2 x i1> poison to i2
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = bitcast <4 x i1> poison to i4
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = bitcast <8 x i1> poison to i8
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = bitcast <16 x i1> poison to i16
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 128 for instruction: %V32 = bitcast <32 x i1> poison to i32
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
+;
+ %V2 = bitcast <2 x i1> poison to i2
+ %V4 = bitcast <4 x i1> poison to i4
+ %V8 = bitcast <8 x i1> poison to i8
+ %V16 = bitcast <16 x i1> poison to i16
+ %V32 = bitcast <32 x i1> poison to i32
+ ret void
+}
+
+define void @unpack() {
+; ALL-LABEL: 'unpack'
+; ALL-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %V2 = bitcast i2 poison to <2 x i1>
+; ALL-NEXT: Cost Model: Found an estimated cost of 12 for instruction: %V4 = bitcast i4 poison to <4 x i1>
+; ALL-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %V8 = bitcast i8 poison to <8 x i1>
+; ALL-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %V16 = bitcast i16 poison to <16 x i1>
+; ALL-NEXT: Cost Model: Found an estimated cost of 96 for instruction: %V32 = bitcast i32 poison to <32 x i1>
+; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret void
+;
+; ALL-SIZE-LABEL: 'unpack'
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 6 for instruction: %V2 = bitcast i2 poison to <2 x i1>
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 12 for instruction: %V4 = bitcast i4 poison to <4 x i1>
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 24 for instruction: %V8 = bitcast i8 poison to <8 x i1>
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 48 for instruction: %V16 = bitcast i16 poison to <16 x i1>
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 96 for instruction: %V32 = bitcast i32 poison to <32 x i1>
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
+;
+ %V2 = bitcast i2 poison to <2 x i1>
+ %V4 = bitcast i4 poison to <4 x i1>
+ %V8 = bitcast i8 poison to <8 x i1>
+ %V16 = bitcast i16 poison to <16 x i1>
+ %V32 = bitcast i32 poison to <32 x i1>
+ ret void
+}
+
+; A single element needs no packing, a bitcast between two vectors of i1 is a
+; move, and a scalable vector keeps the generic cost.
+define void @unchanged() {
+; ALL-LABEL: 'unchanged'
+; ALL-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = bitcast <1 x i1> poison to i1
+; ALL-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %W1 = bitcast i1 poison to <1 x i1>
+; ALL-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %S = bitcast <vscale x 8 x i1> poison to <vscale x 8 x i1>
+; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret void
+;
+; ALL-SIZE-LABEL: 'unchanged'
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = bitcast <1 x i1> poison to i1
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %W1 = bitcast i1 poison to <1 x i1>
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 0 for instruction: %S = bitcast <vscale x 8 x i1> poison to <vscale x 8 x i1>
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
+;
+ %V1 = bitcast <1 x i1> poison to i1
+ %W1 = bitcast i1 poison to <1 x i1>
+ %S = bitcast <vscale x 8 x i1> poison to <vscale x 8 x i1>
+ ret void
+}
diff --git a/llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll b/llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll
new file mode 100644
index 0000000000000..cabfd0b1974af
--- /dev/null
+++ b/llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll
@@ -0,0 +1,57 @@
+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -mtriple=amdgpu9.00-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu10.30-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -mattr=-real-true16 -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -passes="print<cost-model>" -cost-kind=code-size 2>&1 -disable-output | FileCheck -check-prefixes=ALL-SIZE %s
+; END.
+
+; The SLP vectorizer emits an add reduction over a vector of i1 as a bit count
+; over a packed mask. The packing takes 4 instructions per element on every
+; generation from GFX9 to GFX12, so the cost does not depend on the subtarget.
+; An and reduction and an or reduction use the same packing, see reduce-and.ll
+; and reduce-or.ll. The intrinsic itself lowers to a parity chain that costs
+; about 2 per element, and this hook prices that case as the packed form too,
+; because the cost model does not see which form the caller will build.
+
+define void @reduce_add_i1() {
+; ALL-LABEL: 'reduce_add_i1'
+; ALL-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.add.v2i1(<2 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.add.v4i1(<4 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.add.v8i1(<8 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.add.v16i1(<16 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.add.v64i1(<64 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret void
+;
+; ALL-SIZE-LABEL: 'reduce_add_i1'
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.add.v2i1(<2 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.add.v4i1(<4 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.add.v8i1(<8 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.add.v16i1(<16 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.add.v64i1(<64 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
+;
+ %V2 = call i1 @llvm.vector.reduce.add.v2i1(<2 x i1> poison)
+ %V4 = call i1 @llvm.vector.reduce.add.v4i1(<4 x i1> poison)
+ %V8 = call i1 @llvm.vector.reduce.add.v8i1(<8 x i1> poison)
+ %V16 = call i1 @llvm.vector.reduce.add.v16i1(<16 x i1> poison)
+ %V64 = call i1 @llvm.vector.reduce.add.v64i1(<64 x i1> poison)
+ ret void
+}
+
+; One element needs no packing, and a scalable vector keeps the generic cost.
+define void @unchanged(<vscale x 8 x i1> %s) {
+; ALL-LABEL: 'unchanged'
+; ALL-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.add.v1i1(<1 x i1> poison)
+; ALL-NEXT: Cost Model: Invalid cost for instruction: %S = call i1 @llvm.vector.reduce.add.nxv8i1(<vscale x 8 x i1> %s)
+; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret void
+;
+; ALL-SIZE-LABEL: 'unchanged'
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.add.v1i1(<1 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Invalid cost for instruction: %S = call i1 @llvm.vector.reduce.add.nxv8i1(<vscale x 8 x i1> %s)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
+;
+ %V1 = call i1 @llvm.vector.reduce.add.v1i1(<1 x i1> poison)
+ %S = call i1 @llvm.vector.reduce.add.nxv8i1(<vscale x 8 x i1> %s)
+ ret void
+}
diff --git a/llvm/test/Analysis/CostModel/AMDGPU/reduce-and.ll b/llvm/test/Analysis/CostModel/AMDGPU/reduce-and.ll
index 45971bd6f4ba0..daeb790e77c9d 100644
--- a/llvm/test/Analysis/CostModel/AMDGPU/reduce-and.ll
+++ b/llvm/test/Analysis/CostModel/AMDGPU/reduce-and.ll
@@ -6,26 +6,26 @@
define i32 @reduce_i1(i32 %arg) {
; ALL-LABEL: 'reduce_i1'
; ALL-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.and.v1i1(<1 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V2 = call i1 @llvm.vector.reduce.and.v2i1(<2 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 5 for instruction: %V4 = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 9 for instruction: %V8 = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 17 for instruction: %V16 = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %V32 = call i1 @llvm.vector.reduce.and.v32i1(<32 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 65 for instruction: %V64 = call i1 @llvm.vector.reduce.and.v64i1(<64 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 130 for instruction: %V128 = call i1 @llvm.vector.reduce.and.v128i1(<128 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 260 for instruction: %V256 = call i1 @llvm.vector.reduce.and.v256i1(<256 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.and.v2i1(<2 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 128 for instruction: %V32 = call i1 @llvm.vector.reduce.and.v32i1(<32 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.and.v64i1(<64 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 512 for instruction: %V128 = call i1 @llvm.vector.reduce.and.v128i1(<128 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 1024 for instruction: %V256 = call i1 @llvm.vector.reduce.and.v256i1(<256 x i1> undef)
; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret i32 undef
;
; ALL-SIZE-LABEL: 'reduce_i1'
; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.and.v1i1(<1 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V2 = call i1 @llvm.vector.reduce.and.v2i1(<2 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 5 for instruction: %V4 = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 9 for instruction: %V8 = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 17 for instruction: %V16 = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %V32 = call i1 @llvm.vector.reduce.and.v32i1(<32 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 65 for instruction: %V64 = call i1 @llvm.vector.reduce.and.v64i1(<64 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 130 for instruction: %V128 = call i1 @llvm.vector.reduce.and.v128i1(<128 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 260 for instruction: %V256 = call i1 @llvm.vector.reduce.and.v256i1(<256 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.and.v2i1(<2 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.and.v4i1(<4 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.and.v8i1(<8 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.and.v16i1(<16 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 128 for instruction: %V32 = call i1 @llvm.vector.reduce.and.v32i1(<32 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.and.v64i1(<64 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 512 for instruction: %V128 = call i1 @llvm.vector.reduce.and.v128i1(<128 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1024 for instruction: %V256 = call i1 @llvm.vector.reduce.and.v256i1(<256 x i1> undef)
; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret i32 undef
;
%V1 = call i1 @llvm.vector.reduce.and.v1i1(<1 x i1> undef)
diff --git a/llvm/test/Analysis/CostModel/AMDGPU/reduce-or.ll b/llvm/test/Analysis/CostModel/AMDGPU/reduce-or.ll
index e92285c25410a..ae5f08e7283ba 100644
--- a/llvm/test/Analysis/CostModel/AMDGPU/reduce-or.ll
+++ b/llvm/test/Analysis/CostModel/AMDGPU/reduce-or.ll
@@ -6,26 +6,26 @@
define i32 @reduce_i1(i32 %arg) {
; ALL-LABEL: 'reduce_i1'
; ALL-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.or.v1i1(<1 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V2 = call i1 @llvm.vector.reduce.or.v2i1(<2 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 5 for instruction: %V4 = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 9 for instruction: %V8 = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 17 for instruction: %V16 = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %V32 = call i1 @llvm.vector.reduce.or.v32i1(<32 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 65 for instruction: %V64 = call i1 @llvm.vector.reduce.or.v64i1(<64 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 130 for instruction: %V128 = call i1 @llvm.vector.reduce.or.v128i1(<128 x i1> undef)
-; ALL-NEXT: Cost Model: Found an estimated cost of 260 for instruction: %V256 = call i1 @llvm.vector.reduce.or.v256i1(<256 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.or.v2i1(<2 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 128 for instruction: %V32 = call i1 @llvm.vector.reduce.or.v32i1(<32 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.or.v64i1(<64 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 512 for instruction: %V128 = call i1 @llvm.vector.reduce.or.v128i1(<128 x i1> undef)
+; ALL-NEXT: Cost Model: Found an estimated cost of 1024 for instruction: %V256 = call i1 @llvm.vector.reduce.or.v256i1(<256 x i1> undef)
; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret i32 undef
;
; ALL-SIZE-LABEL: 'reduce_i1'
; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.or.v1i1(<1 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 3 for instruction: %V2 = call i1 @llvm.vector.reduce.or.v2i1(<2 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 5 for instruction: %V4 = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 9 for instruction: %V8 = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 17 for instruction: %V16 = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 33 for instruction: %V32 = call i1 @llvm.vector.reduce.or.v32i1(<32 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 65 for instruction: %V64 = call i1 @llvm.vector.reduce.or.v64i1(<64 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 130 for instruction: %V128 = call i1 @llvm.vector.reduce.or.v128i1(<128 x i1> undef)
-; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 260 for instruction: %V256 = call i1 @llvm.vector.reduce.or.v256i1(<256 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.or.v2i1(<2 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.or.v4i1(<4 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.or.v8i1(<8 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.or.v16i1(<16 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 128 for instruction: %V32 = call i1 @llvm.vector.reduce.or.v32i1(<32 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.or.v64i1(<64 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 512 for instruction: %V128 = call i1 @llvm.vector.reduce.or.v128i1(<128 x i1> undef)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1024 for instruction: %V256 = call i1 @llvm.vector.reduce.or.v256i1(<256 x i1> undef)
; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret i32 undef
;
%V1 = call i1 @llvm.vector.reduce.or.v1i1(<1 x i1> undef)
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll
new file mode 100644
index 0000000000000..5649c447c1396
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll
@@ -0,0 +1,109 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes=slp-vectorizer -S -mtriple=amdgpu10.30-amd-amdhsa < %s | FileCheck %s
+; RUN: opt -passes=slp-vectorizer -S -mtriple=amdgpu12.01-amd-amdhsa < %s | FileCheck %s
+; A low threshold makes SLP emit the reduction, so the cost is what keeps it away.
+; RUN: opt -passes=slp-vectorizer -S -mtriple=amdgpu12.01-amd-amdhsa \
+; RUN: -slp-threshold=-100 < %s | FileCheck %s --check-prefix=FORCED
+
+; SLP can turn this chain of adds into a reduction over a vector of i1, and it
+; emits that as a ctpop over a packed mask. The packing costs four instructions
+; per element, so the scalar chain is cheaper.
+
+; This counts how many entries of a table are smaller than a key.
+define i32 @count_smaller(ptr addrspace(3) %tab, i32 %key) {
+; CHECK-LABEL: define i32 @count_smaller(
+; CHECK-SAME: ptr addrspace(3) [[TAB:%.*]], i32 [[KEY:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 1
+; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 2
+; CHECK-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 3
+; CHECK-NEXT: [[P4:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 4
+; CHECK-NEXT: [[P5:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 5
+; CHECK-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 6
+; CHECK-NEXT: [[P7:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 7
+; CHECK-NEXT: [[V0:%.*]] = load i32, ptr addrspace(3) [[TAB]], align 4
+; CHECK-NEXT: [[V1:%.*]] = load i32, ptr addrspace(3) [[P1]], align 4
+; CHECK-NEXT: [[V2:%.*]] = load i32, ptr addrspace(3) [[P2]], align 4
+; CHECK-NEXT: [[V3:%.*]] = load i32, ptr addrspace(3) [[P3]], align 4
+; CHECK-NEXT: [[V4:%.*]] = load i32, ptr addrspace(3) [[P4]], align 4
+; CHECK-NEXT: [[V5:%.*]] = load i32, ptr addrspace(3) [[P5]], align 4
+; CHECK-NEXT: [[V6:%.*]] = load i32, ptr addrspace(3) [[P6]], align 4
+; CHECK-NEXT: [[V7:%.*]] = load i32, ptr addrspace(3) [[P7]], align 4
+; CHECK-NEXT: [[C0:%.*]] = icmp slt i32 [[V0]], [[KEY]]
+; CHECK-NEXT: [[C1:%.*]] = icmp slt i32 [[V1]], [[KEY]]
+; CHECK-NEXT: [[C2:%.*]] = icmp slt i32 [[V2]], [[KEY]]
+; CHECK-NEXT: [[C3:%.*]] = icmp slt i32 [[V3]], [[KEY]]
+; CHECK-NEXT: [[C4:%.*]] = icmp slt i32 [[V4]], [[KEY]]
+; CHECK-NEXT: [[C5:%.*]] = icmp slt i32 [[V5]], [[KEY]]
+; CHECK-NEXT: [[C6:%.*]] = icmp slt i32 [[V6]], [[KEY]]
+; CHECK-NEXT: [[C7:%.*]] = icmp slt i32 [[V7]], [[KEY]]
+; CHECK-NEXT: [[Z0:%.*]] = zext i1 [[C0]] to i32
+; CHECK-NEXT: [[Z1:%.*]] = zext i1 [[C1]] to i32
+; CHECK-NEXT: [[Z2:%.*]] = zext i1 [[C2]] to i32
+; CHECK-NEXT: [[Z3:%.*]] = zext i1 [[C3]] to i32
+; CHECK-NEXT: [[Z4:%.*]] = zext i1 [[C4]] to i32
+; CHECK-NEXT: [[Z5:%.*]] = zext i1 [[C5]] to i32
+; CHECK-NEXT: [[Z6:%.*]] = zext i1 [[C6]] to i32
+; CHECK-NEXT: [[Z7:%.*]] = zext i1 [[C7]] to i32
+; CHECK-NEXT: [[S1:%.*]] = add i32 [[Z0]], [[Z1]]
+; CHECK-NEXT: [[S2:%.*]] = add i32 [[S1]], [[Z2]]
+; CHECK-NEXT: [[S3:%.*]] = add i32 [[S2]], [[Z3]]
+; CHECK-NEXT: [[S4:%.*]] = add i32 [[S3]], [[Z4]]
+; CHECK-NEXT: [[S5:%.*]] = add i32 [[S4]], [[Z5]]
+; CHECK-NEXT: [[S6:%.*]] = add i32 [[S5]], [[Z6]]
+; CHECK-NEXT: [[S7:%.*]] = add i32 [[S6]], [[Z7]]
+; CHECK-NEXT: ret i32 [[S7]]
+;
+; FORCED-LABEL: define i32 @count_smaller(
+; FORCED-SAME: ptr addrspace(3) [[TAB:%.*]], i32 [[KEY:%.*]]) {
+; FORCED-NEXT: [[ENTRY:.*:]]
+; FORCED-NEXT: [[TMP0:%.*]] = load <8 x i32>, ptr addrspace(3) [[TAB]], align 4
+; FORCED-NEXT: [[TMP1:%.*]] = insertelement <8 x i32> poison, i32 [[KEY]], i64 0
+; FORCED-NEXT: [[TMP2:%.*]] = shufflevector <8 x i32> [[TMP1]], <8 x i32> poison, <8 x i32> zeroinitializer
+; FORCED-NEXT: [[TMP3:%.*]] = icmp slt <8 x i32> [[TMP0]], [[TMP2]]
+; FORCED-NEXT: [[TMP4:%.*]] = bitcast <8 x i1> [[TMP3]] to i8
+; FORCED-NEXT: [[TMP5:%.*]] = call i8 @llvm.ctpop.i8(i8 [[TMP4]])
+; FORCED-NEXT: [[TMP6:%.*]] = zext i8 [[TMP5]] to i32
+; FORCED-NEXT: ret i32 [[TMP6]]
+;
+entry:
+ %p1 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 1
+ %p2 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 2
+ %p3 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 3
+ %p4 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 4
+ %p5 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 5
+ %p6 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 6
+ %p7 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 7
+ %v0 = load i32, ptr addrspace(3) %tab, align 4
+ %v1 = load i32, ptr addrspace(3) %p1, align 4
+ %v2 = load i32, ptr addrspace(3) %p2, align 4
+ %v3 = load i32, ptr addrspace(3) %p3, align 4
+ %v4 = load i32, ptr addrspace(3) %p4, align 4
+ %v5 = load i32, ptr addrspace(3) %p5, align 4
+ %v6 = load i32, ptr addrspace(3) %p6, align 4
+ %v7 = load i32, ptr addrspace(3) %p7, align 4
+ %c0 = icmp slt i32 %v0, %key
+ %c1 = icmp slt i32 %v1, %key
+ %c2 = icmp slt i32 %v2, %key
+ %c3 = icmp slt i32 %v3, %key
+ %c4 = icmp slt i32 %v4, %key
+ %c5 = icmp slt i32 %v5, %key
+ %c6 = icmp slt i32 %v6, %key
+ %c7 = icmp slt i32 %v7, %key
+ %z0 = zext i1 %c0 to i32
+ %z1 = zext i1 %c1 to i32
+ %z2 = zext i1 %c2 to i32
+ %z3 = zext i1 %c3 to i32
+ %z4 = zext i1 %c4 to i32
+ %z5 = zext i1 %c5 to i32
+ %z6 = zext i1 %c6 to i32
+ %z7 = zext i1 %c7 to i32
+ %s1 = add i32 %z0, %z1
+ %s2 = add i32 %s1, %z2
+ %s3 = add i32 %s2, %z3
+ %s4 = add i32 %s3, %z4
+ %s5 = add i32 %s4, %z5
+ %s6 = add i32 %s5, %z6
+ %s7 = add i32 %s6, %z7
+ ret i32 %s7
+}
More information about the llvm-commits
mailing list