[llvm] [AMDGPU] Fix performFMACombine FDOT2 fold for subnormal handling (PR #205101)
Wooseok Lee via llvm-commits
llvm-commits at lists.llvm.org
Wed Aug 12 09:08:25 PDT 2026
https://github.com/wooseoklee updated https://github.com/llvm/llvm-project/pull/205101
>From 6892132263da3dc6c235153b2e7796e1207a24c3 Mon Sep 17 00:00:00 2001
From: wooseoklee <wolee at amd.com>
Date: Tue, 16 Jun 2026 12:40:40 -0500
Subject: [PATCH] [AMDGPU] Fix performFMACombine FDOT2 fold for subnormal
handling
The fold from v_fma_mix_f32 pairs to v_dot2_f32_f16/v_dot2c_f32_f16
was gated only on fp-contract flags, ignoring how each instruction
handles f16 subnormal inputs under different denormal modes.
Hardware testing across multiple GPU generations shows that gfx90a
(CDNA2) is the sole outlier: v_dot2c unconditionally flushes f16
subnormal inputs to zero in all MODE configurations, while v_fma_mix_f32
preserves them when ieee=1 (the default compute kernel mode). All other
tested GPUs with dot2 instruction do not flush f16 subnormal inputs.
Add GCNSubtarget::dot2UnconditionalFlush() to capture this hardware
quirk.
Gate the fold on the function's f32 denormal mode:
- dot2UnconditionalFlush(): allow fold only when f32 denorm =
PreserveSign, so both instructions flush f16 subnormals.
- All other GPUs: allow fold only when f32 denorm = IEEE, so both
instructions preserve f16 subnormals. Dynamic mode is also rejected
since the runtime value is unknown.
When both FMAs carry the afn (approximate functions) flag, the caller
has already opted into imprecise results, so any subnormal flushing
introduced by dot2 is acceptable. The denormal mode check is skipped
in that case regardless of GPU or mode.
Update fdot2.ll: add GFX90A-PS and GFX90A-IEEE RUN lines to verify the
gfx90a-specific behavior. Update GFX906/GFX10 checks to reflect that
the fold is now blocked under preserve-sign and allowed under ieee.
---
llvm/lib/Target/AMDGPU/GCNSubtarget.h | 6 ++
llvm/lib/Target/AMDGPU/SIISelLowering.cpp | 35 ++++++++-
llvm/test/CodeGen/AMDGPU/fdot2.ll | 95 +++++++++++++++++++++--
3 files changed, 127 insertions(+), 9 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.h b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
index c42ca8e19ef9c..2bcbef8d46a13 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.h
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
@@ -578,6 +578,12 @@ class GCNSubtarget final : public AMDGPUGenSubtargetInfo,
// if also at the end of the allocation block.
bool hasShift64HighRegBug() const { return HasGFX90AInsts; }
+ // v_dot2c_f32_f16 unconditionally flushes f16 subnormal inputs to zero
+ // regardless of the MODE register, unlike v_fma_mix_f32 which respects it.
+ bool dot2UnconditionalFlush() const {
+ return HasGFX90AInsts && !HasGFX940Insts;
+ }
+
// Has one cycle hazard on transcendental instruction feeding a
// non transcendental VALU.
bool hasTransForwardingHazard() const { return HasGFX940Insts; }
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 939a90b7d2461..a98f60c5809d6 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -18330,9 +18330,38 @@ SDValue SITargetLowering::performFMACombine(SDNode *N,
Op2.getOpcode() != ISD::FP_EXTEND)
return SDValue();
- // fdot2_f32_f16 always flushes fp32 denormal operand and output to zero,
- // regardless of the denorm mode setting. Therefore,
- // fp-contract is sufficient to allow generating fdot2.
+ // The fdot2 fold (fma_mix -> dot2) is only safe when both instructions agree
+ // on how f16 subnormal inputs are handled. However, if both FMAs carry afn
+ // the caller accepts approximate results, so any subnormal flushing
+ // introduced by dot2 is acceptable regardless of mode.
+ //
+ // gfx90a (CDNA2) is the sole exception (dot2UnconditionalFlush): v_dot2c
+ // unconditionally flushes f16 subnormal inputs to zero regardless of MODE,
+ // while v_fma_mix_f32 preserves them when ieee=1 (the default compute kernel
+ // mode). The fold is safe only when f32 denorm = PreserveSign, which implies
+ // ieee=0 so both flush.
+ //
+ // All other GPUs: v_dot2 does NOT flush f16 subnormal inputs. v_fma_mix_f32
+ // flushes them only when f32 denorm = PreserveSign. The fold is safe only
+ // when f32 denorm is IEEE (both preserve the subnormal). Dynamic mode is
+ // also rejected since the runtime value is unknown.
+ bool AllowInaccuracy = N->getFlags().hasApproximateFuncs() &&
+ FMA->getFlags().hasApproximateFuncs();
+ if (!AllowInaccuracy) {
+ const MachineFunction &MF = DAG.getMachineFunction();
+ DenormalMode Mode = MF.getDenormalMode(APFloat::IEEEsingle());
+ if (Subtarget->dot2UnconditionalFlush()) {
+ // gfx90a: fold safe only when f32 denorm flushes.
+ if (Mode != DenormalMode::getPreserveSign())
+ return SDValue();
+ } else {
+ // All other GPUs: fold safe only when f32 denorm is IEEE.
+ if (Mode != DenormalMode::getIEEE())
+ return SDValue();
+ }
+ }
+
+ // fp-contract allows reassociating the fma tree into a dot product.
const TargetOptions &Options = DAG.getTarget().Options;
if (Options.AllowFPOpFusion == FPOpFusion::Fast ||
(N->getFlags().hasAllowContract() &&
diff --git a/llvm/test/CodeGen/AMDGPU/fdot2.ll b/llvm/test/CodeGen/AMDGPU/fdot2.ll
index a8866870e347d..ce04bf760303e 100644
--- a/llvm/test/CodeGen/AMDGPU/fdot2.ll
+++ b/llvm/test/CodeGen/AMDGPU/fdot2.ll
@@ -6,6 +6,10 @@
; RUN: llc -mtriple=amdgpu9.06 -denormal-fp-math=preserve-sign -fp-contract=fast < %s | FileCheck %s -check-prefixes=GCN,GFX906-CONTRACT
; RUN: llc -mtriple=amdgpu9.06 -denormal-fp-math=ieee -fp-contract=fast < %s | FileCheck %s -check-prefixes=GCN,GFX906-DENORM-CONTRACT
; RUN: llc -mtriple=amdgpu9.06 -denormal-fp-math-f32=preserve-sign -mattr="+dot7-insts,-dot10-insts" < %s | FileCheck %s -check-prefixes=GCN,GFX906-DOT10-DISABLED
+; RUN: llc -mtriple=amdgpu9.0a -denormal-fp-math-f32=preserve-sign < %s | FileCheck %s -check-prefixes=GCN,GFX90A-PS
+; RUN: llc -mtriple=amdgpu9.0a -denormal-fp-math-f32=ieee < %s | FileCheck %s -check-prefixes=GCN,GFX90A-IEEE
+; RUN: llc -mtriple=amdgpu9.50 -denormal-fp-math-f32=preserve-sign < %s | FileCheck %s -check-prefixes=GCN,GFX950-DENORM
+; RUN: llc -mtriple=amdgpu9.50 -denormal-fp-math-f32=ieee < %s | FileCheck %s -check-prefixes=GCN,GFX950-IEEE
; (fadd (fmul S1.x, S2.x), (fadd (fmul (S1.y, S2.y), z))) -> (fdot2 S1, S2, z)
; Tests to make sure fdot2 is not generated when vector elements of dot-product expressions
@@ -77,13 +81,15 @@ entry:
; GCN-LABEL: {{^}}dotproduct_f16_f32_contract
-; GFX906-DL-UNSAFE: v_dot2_f32_f16
-; GFX10-DL-UNSAFE: v_dot2c_f32_f16
+; GFX906-DL-UNSAFE: v_fma_mix_f32
+; GFX10-DL-UNSAFE: v_fma_mix_f32
-; GFX906-CONTRACT: v_dot2_f32_f16
+; GFX906-CONTRACT: v_fma_mix_f32
; GFX906-DENORM-CONTRACT: v_dot2_f32_f16
; GFX906-DOT10-DISABLED: v_fma_mix_f32
+; GFX90A-PS: v_dot2c_f32_f16
+; GFX90A-IEEE: v_fma_mix_f32
define amdgpu_kernel void @dotproduct_f16_f32_contract(ptr addrspace(1) %src1,
ptr addrspace(1) %src2,
ptr addrspace(1) nocapture %dst) {
@@ -149,12 +155,14 @@ entry:
; - "dot10-insts" is enabled
; GCN-LABEL: {{^}}dotproduct_diffvecorder_contract
-; GFX906-DL-UNSAFE: v_dot2_f32_f16
-; GFX10-DL-UNSAFE: v_dot2c_f32_f16
+; GFX906-DL-UNSAFE: v_fma_mix_f32
+; GFX10-DL-UNSAFE: v_fma_mix_f32
-; GFX906-CONTRACT: v_dot2_f32_f16
+; GFX906-CONTRACT: v_fma_mix_f32
; GFX906-DENORM-CONTRACT: v_dot2_f32_f16
; GFX906-DOT10-DISABLED: v_fma_mix_f32
+; GFX90A-PS: v_dot2c_f32_f16
+; GFX90A-IEEE: v_fma_mix_f32
define amdgpu_kernel void @dotproduct_diffvecorder_contract(ptr addrspace(1) %src1,
ptr addrspace(1) %src2,
ptr addrspace(1) nocapture %dst) {
@@ -411,3 +419,78 @@ entry:
store float %acc2, ptr addrspace(1) %dst, align 4
ret void
}
+
+; Fold is suppressed with f32 denorm = preserve-sign: on gfx950, v_dot2 does
+; not flush f16 subnormal inputs but v_fma_mix_f32 would, so they disagree.
+; GCN-LABEL: {{^}}dotproduct_f16_f32_contract_ieee_denorm
+; GFX950-DENORM: v_fma_mix_f32
+; GFX950-DENORM: v_fma_mix_f32
+; GFX950-DENORM-NOT: v_dot2c_f32_f16
+; GFX950-DENORM-NOT: v_dot2_f32_f16
+define amdgpu_kernel void @dotproduct_f16_f32_contract_ieee_denorm(<2 x half> %a, <2 x half> %b, float %z, ptr addrspace(1) %out) {
+ %ax = extractelement <2 x half> %a, i32 0
+ %axf = fpext half %ax to float
+ %ay = extractelement <2 x half> %a, i32 1
+ %ayf = fpext half %ay to float
+
+ %bx = extractelement <2 x half> %b, i32 0
+ %bxf = fpext half %bx to float
+ %by = extractelement <2 x half> %b, i32 1
+ %byf = fpext half %by to float
+
+ %inner = call contract float @llvm.fma.f32(float %ayf, float %byf, float %z)
+ %outer = call contract float @llvm.fma.f32(float %axf, float %bxf, float %inner)
+ store float %outer, ptr addrspace(1) %out
+ ret void
+}
+
+; Dynamic denormal mode: compile-time mode unknown, conservatively suppress fold.
+; GCN-LABEL: {{^}}dotproduct_f16_f32_contract_dynamic_denorm
+; GFX950-DENORM: v_fma_mix_f32
+; GFX950-DENORM: v_fma_mix_f32
+; GFX950-DENORM-NOT: v_dot2c_f32_f16
+; GFX950-DENORM-NOT: v_dot2_f32_f16
+define amdgpu_kernel void @dotproduct_f16_f32_contract_dynamic_denorm(<2 x half> %a, <2 x half> %b, float %z, ptr addrspace(1) %out) #0 {
+ %ax = extractelement <2 x half> %a, i32 0
+ %axf = fpext half %ax to float
+ %ay = extractelement <2 x half> %a, i32 1
+ %ayf = fpext half %ay to float
+
+ %bx = extractelement <2 x half> %b, i32 0
+ %bxf = fpext half %bx to float
+ %by = extractelement <2 x half> %b, i32 1
+ %byf = fpext half %by to float
+
+ %inner = call contract float @llvm.fma.f32(float %ayf, float %byf, float %z)
+ %outer = call contract float @llvm.fma.f32(float %axf, float %bxf, float %inner)
+ store float %outer, ptr addrspace(1) %out
+ ret void
+}
+
+; afn on both FMAs overrides the denormal gating: the caller accepts approximate
+; results, so dot2's subnormal flushing is acceptable regardless of mode or GPU.
+; GCN-LABEL: {{^}}dotproduct_f16_f32_afn
+; GFX906-CONTRACT: v_dot2_f32_f16
+; GFX906-DENORM-CONTRACT: v_dot2_f32_f16
+; GFX90A-PS: v_dot2c_f32_f16
+; GFX90A-IEEE: v_dot2c_f32_f16
+; GFX950-DENORM: v_dot2c_f32_f16
+; GFX950-IEEE: v_dot2c_f32_f16
+define amdgpu_kernel void @dotproduct_f16_f32_afn(<2 x half> %a, <2 x half> %b, float %z, ptr addrspace(1) %out) {
+ %ax = extractelement <2 x half> %a, i32 0
+ %axf = fpext half %ax to float
+ %ay = extractelement <2 x half> %a, i32 1
+ %ayf = fpext half %ay to float
+
+ %bx = extractelement <2 x half> %b, i32 0
+ %bxf = fpext half %bx to float
+ %by = extractelement <2 x half> %b, i32 1
+ %byf = fpext half %by to float
+
+ %inner = call afn contract float @llvm.fma.f32(float %ayf, float %byf, float %z)
+ %outer = call afn contract float @llvm.fma.f32(float %axf, float %bxf, float %inner)
+ store float %outer, ptr addrspace(1) %out
+ ret void
+}
+
+attributes #0 = { denormal_fpenv(float: dynamic) }
More information about the llvm-commits
mailing list