[llvm] [AMDGPU] Price an add reduction over a vector of i1 (PR #217327)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Aug 19 05:34:17 PDT 2026
https://github.com/michaelselehov created https://github.com/llvm/llvm-project/pull/217327
The SLP vectorizer emits an unsigned add reduction over <n x i1> as ctpop(bitcast <n x i1> to in). Every element of the vector is a separate mask, so the bitcast has to pack them: the target selects a 0 or a 1 per element, shifts it into place and merges it. The generic model prices the reduction as a shuffle tree, which is much cheaper than the packing. The vectorizer then replaces a cheap scalar chain of adds with the packed form.
This override charges 4 instructions per element. Instruction counts after codegen give 4.1 to 4.3 per element on gfx900, gfx942, gfx1030, gfx1100 and gfx1201, at 8 and at 16 elements. The counts use a divergent operand, and they exclude a baseline of the same loads and compares. They also exclude s_delay_alu and s_wait_alu, which GFX11 and GFX12 need between dependent operations and which no cost model counts. The number of real instructions does not depend on the subtarget, so neither does the cost.
A reduction that keeps the i1 result is a parity computation, and it needs about 2 instructions per element. This hook prices that case as the packed form as well, because the cost model does not see how the result is used. The vectorizer emits the packed form whenever the result is wider than i1.
Assisted-by: Claude Opus
>From 2f63c9c336d992d12a710a506282f7493e4d0645 Mon Sep 17 00:00:00 2001
From: Michael Selehov <michael.selehov at amd.com>
Date: Fri, 14 Aug 2026 09:35:39 -0500
Subject: [PATCH] [AMDGPU] Price an add reduction over a vector of i1
The SLP vectorizer emits an unsigned add reduction over <n x i1> as
ctpop(bitcast <n x i1> to in). Every element of the vector is a separate mask,
so the bitcast has to pack them: the target selects a 0 or a 1 per element,
shifts it into place and merges it. The generic model prices the reduction as a
shuffle tree, which is much cheaper than the packing. The vectorizer then
replaces a cheap scalar chain of adds with the packed form.
This override charges 4 instructions per element. Instruction counts after
codegen give 4.1 to 4.3 per element on gfx900, gfx942, gfx1030, gfx1100 and
gfx1201, at 8 and at 16 elements. The counts use a divergent operand, and they
exclude a baseline of the same loads and compares. They also exclude s_delay_alu
and s_wait_alu, which GFX11 and GFX12 need between dependent operations and
which no cost model counts. The number of real instructions does not depend on
the subtarget, so neither does the cost.
A reduction that keeps the i1 result is a parity computation, and it needs about
2 instructions per element. This hook prices that case as the packed form as
well, because the cost model does not see how the result is used. The
vectorizer emits the packed form whenever the result is wider than i1.
Assisted-by: Claude Opus
---
.../AMDGPU/AMDGPUTargetTransformInfo.cpp | 12 ++
.../CostModel/AMDGPU/reduce-add-i1.ll | 54 +++++++++
.../AMDGPU/reduction-i1-ctpop-cost.ll | 109 ++++++++++++++++++
3 files changed, 175 insertions(+)
create mode 100644 llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll
create mode 100644 llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
index 980e26082064f..ce79234dabfc0 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
@@ -987,6 +987,18 @@ GCNTTIImpl::getArithmeticReductionCost(unsigned Opcode, VectorType *Ty,
if (TTI::requiresOrderedReduction(FMF))
return BaseT::getArithmeticReductionCost(Opcode, Ty, FMF, CostKind);
+ // An add reduction over a vector of i1 is lowered as
+ // ctpop(bitcast <n x i1> to in). Every element is a separate mask, so the
+ // bitcast packs them with a select, a shift and an or per element. That takes
+ // 4 instructions per element from GFX9 to GFX12. The generic model prices the
+ // reduction as a shuffle tree, which is far too cheap.
+ if (Opcode == Instruction::Add && Ty->getElementType()->isIntegerTy(1)) {
+ if (auto *FVT = dyn_cast<FixedVectorType>(Ty);
+ FVT && FVT->getNumElements() > 1)
+ return InstructionCost(4) * FVT->getNumElements() *
+ getFullRateInstrCost();
+ }
+
EVT OrigTy = TLI->getValueType(DL, Ty);
// Computes cost on targets that have packed math instructions(which support
diff --git a/llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll b/llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll
new file mode 100644
index 0000000000000..af2bbae707070
--- /dev/null
+++ b/llvm/test/Analysis/CostModel/AMDGPU/reduce-add-i1.ll
@@ -0,0 +1,54 @@
+; NOTE: Assertions have been autogenerated by utils/update_analyze_test_checks.py UTC_ARGS: --version 6
+; RUN: opt < %s -mtriple=amdgpu9.00-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu10.30-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -mattr=-real-true16 -passes="print<cost-model>" -cost-kind=throughput 2>&1 -disable-output | FileCheck -check-prefixes=ALL %s
+; RUN: opt < %s -mtriple=amdgpu12.01-unknown-amdhsa -passes="print<cost-model>" -cost-kind=code-size 2>&1 -disable-output | FileCheck -check-prefixes=ALL-SIZE %s
+; END.
+
+; An add reduction over a vector of i1 is lowered as ctpop(bitcast <n x i1> to
+; in). The bitcast packs the elements, because every element is a separate mask.
+; The packing takes 4 instructions per element on every generation from GFX9 to
+; GFX12, so the cost does not depend on the subtarget.
+
+define void @reduce_add_i1() {
+; ALL-LABEL: 'reduce_add_i1'
+; ALL-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.add.v2i1(<2 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.add.v4i1(<4 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.add.v8i1(<8 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.add.v16i1(<16 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.add.v64i1(<64 x i1> poison)
+; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret void
+;
+; ALL-SIZE-LABEL: 'reduce_add_i1'
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 8 for instruction: %V2 = call i1 @llvm.vector.reduce.add.v2i1(<2 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 16 for instruction: %V4 = call i1 @llvm.vector.reduce.add.v4i1(<4 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 32 for instruction: %V8 = call i1 @llvm.vector.reduce.add.v8i1(<8 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 64 for instruction: %V16 = call i1 @llvm.vector.reduce.add.v16i1(<16 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 256 for instruction: %V64 = call i1 @llvm.vector.reduce.add.v64i1(<64 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
+;
+ %V2 = call i1 @llvm.vector.reduce.add.v2i1(<2 x i1> poison)
+ %V4 = call i1 @llvm.vector.reduce.add.v4i1(<4 x i1> poison)
+ %V8 = call i1 @llvm.vector.reduce.add.v8i1(<8 x i1> poison)
+ %V16 = call i1 @llvm.vector.reduce.add.v16i1(<16 x i1> poison)
+ %V64 = call i1 @llvm.vector.reduce.add.v64i1(<64 x i1> poison)
+ ret void
+}
+
+; One element needs no packing, and a scalable vector keeps the generic cost.
+define void @unchanged(<vscale x 8 x i1> %s) {
+; ALL-LABEL: 'unchanged'
+; ALL-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.add.v1i1(<1 x i1> poison)
+; ALL-NEXT: Cost Model: Invalid cost for instruction: %S = call i1 @llvm.vector.reduce.add.nxv8i1(<vscale x 8 x i1> %s)
+; ALL-NEXT: Cost Model: Found an estimated cost of 10 for instruction: ret void
+;
+; ALL-SIZE-LABEL: 'unchanged'
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: %V1 = call i1 @llvm.vector.reduce.add.v1i1(<1 x i1> poison)
+; ALL-SIZE-NEXT: Cost Model: Invalid cost for instruction: %S = call i1 @llvm.vector.reduce.add.nxv8i1(<vscale x 8 x i1> %s)
+; ALL-SIZE-NEXT: Cost Model: Found an estimated cost of 1 for instruction: ret void
+;
+ %V1 = call i1 @llvm.vector.reduce.add.v1i1(<1 x i1> poison)
+ %S = call i1 @llvm.vector.reduce.add.nxv8i1(<vscale x 8 x i1> %s)
+ ret void
+}
diff --git a/llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll b/llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll
new file mode 100644
index 0000000000000..b5cdfe57d9e91
--- /dev/null
+++ b/llvm/test/Transforms/SLPVectorizer/AMDGPU/reduction-i1-ctpop-cost.ll
@@ -0,0 +1,109 @@
+; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 6
+; RUN: opt -passes=slp-vectorizer -S -mtriple=amdgpu10.30-amd-amdhsa < %s | FileCheck %s
+; RUN: opt -passes=slp-vectorizer -S -mtriple=amdgpu12.01-amd-amdhsa < %s | FileCheck %s
+; A low threshold makes SLP emit the reduction, so the cost is what keeps it away.
+; RUN: opt -passes=slp-vectorizer -S -mtriple=amdgpu12.01-amd-amdhsa \
+; RUN: -slp-threshold=-100 < %s | FileCheck %s --check-prefix=FORCED
+
+; SLP can turn this chain of adds into a reduction over a vector of i1, and it
+; emits that as a ctpop over a packed mask. The packing costs four to six
+; instructions per element here, so the scalar chain is cheaper.
+
+; This counts how many entries of a table are smaller than a key.
+define i32 @count_smaller(ptr addrspace(3) %tab, i32 %key) {
+; CHECK-LABEL: define i32 @count_smaller(
+; CHECK-SAME: ptr addrspace(3) [[TAB:%.*]], i32 [[KEY:%.*]]) {
+; CHECK-NEXT: [[ENTRY:.*:]]
+; CHECK-NEXT: [[P1:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 1
+; CHECK-NEXT: [[P2:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 2
+; CHECK-NEXT: [[P3:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 3
+; CHECK-NEXT: [[P4:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 4
+; CHECK-NEXT: [[P5:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 5
+; CHECK-NEXT: [[P6:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 6
+; CHECK-NEXT: [[P7:%.*]] = getelementptr inbounds i32, ptr addrspace(3) [[TAB]], i32 7
+; CHECK-NEXT: [[V0:%.*]] = load i32, ptr addrspace(3) [[TAB]], align 4
+; CHECK-NEXT: [[V1:%.*]] = load i32, ptr addrspace(3) [[P1]], align 4
+; CHECK-NEXT: [[V2:%.*]] = load i32, ptr addrspace(3) [[P2]], align 4
+; CHECK-NEXT: [[V3:%.*]] = load i32, ptr addrspace(3) [[P3]], align 4
+; CHECK-NEXT: [[V4:%.*]] = load i32, ptr addrspace(3) [[P4]], align 4
+; CHECK-NEXT: [[V5:%.*]] = load i32, ptr addrspace(3) [[P5]], align 4
+; CHECK-NEXT: [[V6:%.*]] = load i32, ptr addrspace(3) [[P6]], align 4
+; CHECK-NEXT: [[V7:%.*]] = load i32, ptr addrspace(3) [[P7]], align 4
+; CHECK-NEXT: [[C0:%.*]] = icmp slt i32 [[V0]], [[KEY]]
+; CHECK-NEXT: [[C1:%.*]] = icmp slt i32 [[V1]], [[KEY]]
+; CHECK-NEXT: [[C2:%.*]] = icmp slt i32 [[V2]], [[KEY]]
+; CHECK-NEXT: [[C3:%.*]] = icmp slt i32 [[V3]], [[KEY]]
+; CHECK-NEXT: [[C4:%.*]] = icmp slt i32 [[V4]], [[KEY]]
+; CHECK-NEXT: [[C5:%.*]] = icmp slt i32 [[V5]], [[KEY]]
+; CHECK-NEXT: [[C6:%.*]] = icmp slt i32 [[V6]], [[KEY]]
+; CHECK-NEXT: [[C7:%.*]] = icmp slt i32 [[V7]], [[KEY]]
+; CHECK-NEXT: [[Z0:%.*]] = zext i1 [[C0]] to i32
+; CHECK-NEXT: [[Z1:%.*]] = zext i1 [[C1]] to i32
+; CHECK-NEXT: [[Z2:%.*]] = zext i1 [[C2]] to i32
+; CHECK-NEXT: [[Z3:%.*]] = zext i1 [[C3]] to i32
+; CHECK-NEXT: [[Z4:%.*]] = zext i1 [[C4]] to i32
+; CHECK-NEXT: [[Z5:%.*]] = zext i1 [[C5]] to i32
+; CHECK-NEXT: [[Z6:%.*]] = zext i1 [[C6]] to i32
+; CHECK-NEXT: [[Z7:%.*]] = zext i1 [[C7]] to i32
+; CHECK-NEXT: [[S1:%.*]] = add i32 [[Z0]], [[Z1]]
+; CHECK-NEXT: [[S2:%.*]] = add i32 [[S1]], [[Z2]]
+; CHECK-NEXT: [[S3:%.*]] = add i32 [[S2]], [[Z3]]
+; CHECK-NEXT: [[S4:%.*]] = add i32 [[S3]], [[Z4]]
+; CHECK-NEXT: [[S5:%.*]] = add i32 [[S4]], [[Z5]]
+; CHECK-NEXT: [[S6:%.*]] = add i32 [[S5]], [[Z6]]
+; CHECK-NEXT: [[S7:%.*]] = add i32 [[S6]], [[Z7]]
+; CHECK-NEXT: ret i32 [[S7]]
+;
+; FORCED-LABEL: define i32 @count_smaller(
+; FORCED-SAME: ptr addrspace(3) [[TAB:%.*]], i32 [[KEY:%.*]]) {
+; FORCED-NEXT: [[ENTRY:.*:]]
+; FORCED-NEXT: [[TMP0:%.*]] = load <8 x i32>, ptr addrspace(3) [[TAB]], align 4
+; FORCED-NEXT: [[TMP1:%.*]] = insertelement <8 x i32> poison, i32 [[KEY]], i64 0
+; FORCED-NEXT: [[TMP2:%.*]] = shufflevector <8 x i32> [[TMP1]], <8 x i32> poison, <8 x i32> zeroinitializer
+; FORCED-NEXT: [[TMP3:%.*]] = icmp slt <8 x i32> [[TMP0]], [[TMP2]]
+; FORCED-NEXT: [[TMP4:%.*]] = bitcast <8 x i1> [[TMP3]] to i8
+; FORCED-NEXT: [[TMP5:%.*]] = call i8 @llvm.ctpop.i8(i8 [[TMP4]])
+; FORCED-NEXT: [[TMP6:%.*]] = zext i8 [[TMP5]] to i32
+; FORCED-NEXT: ret i32 [[TMP6]]
+;
+entry:
+ %p1 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 1
+ %p2 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 2
+ %p3 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 3
+ %p4 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 4
+ %p5 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 5
+ %p6 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 6
+ %p7 = getelementptr inbounds i32, ptr addrspace(3) %tab, i32 7
+ %v0 = load i32, ptr addrspace(3) %tab, align 4
+ %v1 = load i32, ptr addrspace(3) %p1, align 4
+ %v2 = load i32, ptr addrspace(3) %p2, align 4
+ %v3 = load i32, ptr addrspace(3) %p3, align 4
+ %v4 = load i32, ptr addrspace(3) %p4, align 4
+ %v5 = load i32, ptr addrspace(3) %p5, align 4
+ %v6 = load i32, ptr addrspace(3) %p6, align 4
+ %v7 = load i32, ptr addrspace(3) %p7, align 4
+ %c0 = icmp slt i32 %v0, %key
+ %c1 = icmp slt i32 %v1, %key
+ %c2 = icmp slt i32 %v2, %key
+ %c3 = icmp slt i32 %v3, %key
+ %c4 = icmp slt i32 %v4, %key
+ %c5 = icmp slt i32 %v5, %key
+ %c6 = icmp slt i32 %v6, %key
+ %c7 = icmp slt i32 %v7, %key
+ %z0 = zext i1 %c0 to i32
+ %z1 = zext i1 %c1 to i32
+ %z2 = zext i1 %c2 to i32
+ %z3 = zext i1 %c3 to i32
+ %z4 = zext i1 %c4 to i32
+ %z5 = zext i1 %c5 to i32
+ %z6 = zext i1 %c6 to i32
+ %z7 = zext i1 %c7 to i32
+ %s1 = add i32 %z0, %z1
+ %s2 = add i32 %s1, %z2
+ %s3 = add i32 %s2, %z3
+ %s4 = add i32 %s3, %z4
+ %s5 = add i32 %s4, %z5
+ %s6 = add i32 %s5, %z6
+ %s7 = add i32 %s6, %z7
+ ret i32 %s7
+}
More information about the llvm-commits
mailing list