[llvm] [AMDGPU] Widen MUBUF/MTBUF source-vgpr WAR hazard on gfx940-family to SGPR soffset (PR #197267)
Sanket Pandit via llvm-commits
llvm-commits at lists.llvm.org
Sun May 31 19:46:15 PDT 2026
https://github.com/panditsa updated https://github.com/llvm/llvm-project/pull/197267
>From e42537cff4e368caf691a211066b411714c2bdb7 Mon Sep 17 00:00:00 2001
From: Sanket Pandit <sanket.pandit at amd.com>
Date: Thu, 7 May 2026 21:07:21 +0000
Subject: [PATCH 1/4] [AMDGPU] Widen MUBUF/MTBUF source-vgpr WAR hazard on
gfx940-family to SGPR soffset
createsVALUHazard previously gated the MUBUF/MTBUF source-vgpr WAR
hazard to fire only when SOFFSET was a literal or absent. On
gfx940-family subtargets that gate is too narrow: the hazard also fires
when SOFFSET is sourced from an SGPR.
Concretely, on gfx950 a sequence of the form
buffer_store_dwordx4 v[X:X+3], voff, descr, sN offen
v_pk_mul_f32 v[X:X+1], <src>, <src> # next VALU cycle
deterministically commits the post-pk_mul value of v[X+1] to memory for
the second dword of the store; the other three dwords store correctly.
The wait-state window depends on the SOFFSET shape:
- sgpr-sourced SOFFSET: 1 wait state (S_NOP 0)
- literal/absent SOFFSET: 2 wait states (S_NOP 1)
- pre-gfx940 literal SOFFSET (unchanged): 1 wait state
- pre-gfx940 sgpr SOFFSET (unchanged): no hazard
Implementation widens createsVALUHazard's trigger on gfx940-family and
splits checkVALUHazardsHelper's single 2-wait-state scan into a per-MI
window: a 1-cycle scan and a 2-cycle scan, taking the max. FLAT-wide
stores keep the existing 2-wait-state behavior on gfx940 family.
Empirically reproduced and characterised on AMD Instinct MI350X
(gfx950) by Triton's fused-attention backward kernel; the threshold
sweep in repro/minimal_hazard pins the sgpr-SOFFSET window at exactly
1 cycle (a single S_NOP 0 clears the corruption). gfx942 (MI300) was
not validated on hardware; the same gate is applied there because the
existing recognizer rule treats the gfx940 family uniformly.
New MIR test covers literal and SGPR soffset on gfx900 (older), gfx942
and gfx950, plus negative cases (dwordx2 store, non-overlapping write).
Assisted-by: Cursor <cursoragent at cursor.com>
Signed-off-by: Sanket Pandit <sanket.pandit at amd.com>
Co-authored-by: Cursor <cursoragent at cursor.com>
---
.../lib/Target/AMDGPU/GCNHazardRecognizer.cpp | 71 +++++++---
...tore-dwordx4-vpk-mul-war-hazard-gfx942.mir | 122 ++++++++++++++++++
2 files changed, 172 insertions(+), 21 deletions(-)
create mode 100644 llvm/test/CodeGen/AMDGPU/buffer-store-dwordx4-vpk-mul-war-hazard-gfx942.mir
diff --git a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
index 8ab90d953200e..ad3b40fee440a 100644
--- a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
@@ -913,15 +913,18 @@ int GCNHazardRecognizer::createsVALUHazard(const MachineInstr &MI) const {
// (like wbinvl1)
if (VDataIdx == -1)
return -1;
- // For MUBUF/MTBUF instructions this hazard only exists if the
- // instruction is not using a register in the soffset field.
- const MachineOperand *SOffset =
- TII->getNamedOperand(MI, AMDGPU::OpName::soffset);
- // If we have no soffset operand, then assume this field has been
- // hardcoded to zero.
- if (AMDGPU::getRegBitWidth(VDataRCID) > 64 &&
- (!SOffset || !SOffset->isReg()))
- return VDataIdx;
+ if (AMDGPU::getRegBitWidth(VDataRCID) > 64) {
+ // On gfx940-family the BUFFER_STORE source-vgpr WAR hazard exists for
+ // every SOFFSET shape; the wait-state count differs by SOFFSET, and is
+ // computed in checkVALUHazardsHelper. Pre-gfx940 keeps the original
+ // narrow gate.
+ if (ST.hasGFX940Insts())
+ return VDataIdx;
+ const MachineOperand *SOffset =
+ TII->getNamedOperand(MI, AMDGPU::OpName::soffset);
+ if (!SOffset || !SOffset->isReg())
+ return VDataIdx;
+ }
}
// MIMG instructions create a hazard if they don't use a 256-bit T# and
@@ -949,25 +952,51 @@ int GCNHazardRecognizer::createsVALUHazard(const MachineInstr &MI) const {
int GCNHazardRecognizer::checkVALUHazardsHelper(
const MachineOperand &Def, const MachineRegisterInfo &MRI) const {
- // Helper to check for the hazard where VMEM instructions that store more than
- // 8 bytes can have there store data over written by the next instruction.
+ // Helper to check for the hazard where VMEM instructions that store more
+ // than 8 bytes can have their store data overwritten by the next
+ // instruction. On gfx940-family the window depends on the producer's
+ // SOFFSET shape:
+ // - MUBUF/MTBUF wide store with sgpr SOFFSET: 1 wait state.
+ // - MUBUF/MTBUF wide store with literal/absent SOFFSET, and FLAT wide
+ // store: 2 wait states.
+ // Pre-gfx940 keeps a single 1-wait-state window. The 1-cycle sgpr-SOFFSET
+ // window was measured on gfx950 (MI350X); the same gate is applied to the
+ // rest of the gfx940 family to match the existing rule's granularity.
const SIRegisterInfo *TRI = ST.getRegisterInfo();
+ const SIInstrInfo *TII = ST.getInstrInfo();
- const int VALUWaitStates = ST.hasGFX940Insts() ? 2 : 1;
int WaitStatesNeeded = 0;
-
if (!TRI->isVectorRegister(MRI, Def.getReg()))
return WaitStatesNeeded;
- Register Reg = Def.getReg();
- auto IsHazardFn = [this, Reg, TRI](const MachineInstr &MI) {
- int DataIdx = createsVALUHazard(MI);
- return DataIdx >= 0 &&
- TRI->regsOverlap(MI.getOperand(DataIdx).getReg(), Reg);
+ const Register Reg = Def.getReg();
+
+ const int MaxWaitStates = ST.hasGFX940Insts() ? 2 : 1;
+
+ auto WindowFor = [this, TII](const MachineInstr &MI) -> int {
+ if (!ST.hasGFX940Insts())
+ return 1;
+ if (TII->isBUF(MI)) {
+ const MachineOperand *SOffset =
+ TII->getNamedOperand(MI, AMDGPU::OpName::soffset);
+ if (SOffset && SOffset->isReg())
+ return 1;
+ }
+ return 2;
};
- int WaitStatesNeededForDef =
- VALUWaitStates - getWaitStatesSince(IsHazardFn, VALUWaitStates);
- WaitStatesNeeded = std::max(WaitStatesNeeded, WaitStatesNeededForDef);
+ for (int Window = 1; Window <= MaxWaitStates; ++Window) {
+ auto IsHazardFn = [this, Reg, TRI, Window,
+ &WindowFor](const MachineInstr &MI) {
+ int DataIdx = createsVALUHazard(MI);
+ if (DataIdx < 0)
+ return false;
+ if (WindowFor(MI) != Window)
+ return false;
+ return TRI->regsOverlap(MI.getOperand(DataIdx).getReg(), Reg);
+ };
+ int Elapsed = getWaitStatesSince(IsHazardFn, Window);
+ WaitStatesNeeded = std::max(WaitStatesNeeded, Window - Elapsed);
+ }
return WaitStatesNeeded;
}
diff --git a/llvm/test/CodeGen/AMDGPU/buffer-store-dwordx4-vpk-mul-war-hazard-gfx942.mir b/llvm/test/CodeGen/AMDGPU/buffer-store-dwordx4-vpk-mul-war-hazard-gfx942.mir
new file mode 100644
index 0000000000000..db8b47f5cab5f
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/buffer-store-dwordx4-vpk-mul-war-hazard-gfx942.mir
@@ -0,0 +1,122 @@
+# NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
+# RUN: llc -mtriple=amdgcn -mcpu=gfx950 -run-pass=post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,GFX950 %s
+# RUN: llc -mtriple=amdgcn -mcpu=gfx942 -run-pass=post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,GFX942 %s
+# RUN: llc -mtriple=amdgcn -mcpu=gfx900 -run-pass=post-RA-hazard-rec %s -o - | FileCheck -check-prefixes=GCN,GFX900 %s
+
+# Tests for the BUFFER_STORE source-vgpr WAR hazard on gfx940-family
+# subtargets (gfx942, gfx950).
+#
+# A buffer_store_dwordx4 (or any MUBUF store wider than 8 bytes) immediately
+# followed by a VALU instruction that overwrites two of the source vgprs
+# (typically v_pk_mul_f32 / v_pk_add_f32 writing v[X:X+1]) loses the bytes
+# of one dword to the post-write value on gfx940 family. The required
+# wait-state window depends on the producer's SOFFSET shape:
+# - SGPR-sourced SOFFSET: 1 wait state -> S_NOP 0
+# - literal/absent SOFFSET: 2 wait states -> S_NOP 1
+# Empirical measurement on gfx950 (MI350X); the same rule is applied to
+# the rest of the gfx940 family to match the existing recognizer's
+# granularity.
+#
+# Pre-gfx940 the hazard existed only with literal/absent SOFFSET; the
+# pre-existing rule keeps its narrow gating for those targets.
+
+---
+name: buffer_store_dwordx4_sgpr_soffset_then_vpk_mul
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX950-LABEL: name: buffer_store_dwordx4_sgpr_soffset_then_vpk_mul
+ ; GFX950: liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX950-NEXT: {{ $}}
+ ; GFX950-NEXT: BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ ; GFX950-NEXT: S_NOP 0
+ ; GFX950-NEXT: $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GFX950-NEXT: S_ENDPGM 0
+ ;
+ ; GFX942-LABEL: name: buffer_store_dwordx4_sgpr_soffset_then_vpk_mul
+ ; GFX942: liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX942-NEXT: {{ $}}
+ ; GFX942-NEXT: BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ ; GFX942-NEXT: S_NOP 0
+ ; GFX942-NEXT: $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GFX942-NEXT: S_ENDPGM 0
+ ;
+ ; GFX900-LABEL: name: buffer_store_dwordx4_sgpr_soffset_then_vpk_mul
+ ; GFX900: liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX900-NEXT: {{ $}}
+ ; GFX900-NEXT: BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ ; GFX900-NEXT: $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GFX900-NEXT: S_ENDPGM 0
+ BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ S_ENDPGM 0
+...
+
+---
+name: buffer_store_dwordx4_literal_soffset_then_vpk_mul
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX950-LABEL: name: buffer_store_dwordx4_literal_soffset_then_vpk_mul
+ ; GFX950: liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX950-NEXT: {{ $}}
+ ; GFX950-NEXT: BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec
+ ; GFX950-NEXT: S_NOP 1
+ ; GFX950-NEXT: $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GFX950-NEXT: S_ENDPGM 0
+ ;
+ ; GFX942-LABEL: name: buffer_store_dwordx4_literal_soffset_then_vpk_mul
+ ; GFX942: liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX942-NEXT: {{ $}}
+ ; GFX942-NEXT: BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec
+ ; GFX942-NEXT: S_NOP 1
+ ; GFX942-NEXT: $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GFX942-NEXT: S_ENDPGM 0
+ ;
+ ; GFX900-LABEL: name: buffer_store_dwordx4_literal_soffset_then_vpk_mul
+ ; GFX900: liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GFX900-NEXT: {{ $}}
+ ; GFX900-NEXT: BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec
+ ; GFX900-NEXT: S_NOP 0
+ ; GFX900-NEXT: $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GFX900-NEXT: S_ENDPGM 0
+ BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, 0, 0, 0, 0, implicit $exec
+ $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ S_ENDPGM 0
+...
+
+---
+name: buffer_store_dwordx2_sgpr_soffset_then_vpk_mul_no_hazard
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GCN-LABEL: name: buffer_store_dwordx2_sgpr_soffset_then_vpk_mul_no_hazard
+ ; GCN: liveins: $vgpr0_vgpr1, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GCN-NEXT: {{ $}}
+ ; GCN-NEXT: BUFFER_STORE_DWORDX2_OFFEN_exact $vgpr0_vgpr1, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ ; GCN-NEXT: $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GCN-NEXT: S_ENDPGM 0
+ BUFFER_STORE_DWORDX2_OFFEN_exact $vgpr0_vgpr1, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ $vgpr0_vgpr1 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ S_ENDPGM 0
+...
+
+---
+name: buffer_store_dwordx4_sgpr_soffset_then_unrelated_write
+tracksRegLiveness: true
+body: |
+ bb.0:
+ liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GCN-LABEL: name: buffer_store_dwordx4_sgpr_soffset_then_unrelated_write
+ ; GCN: liveins: $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $vgpr8_vgpr9, $sgpr0_sgpr1, $sgpr1, $sgpr8_sgpr9_sgpr10_sgpr11
+ ; GCN-NEXT: {{ $}}
+ ; GCN-NEXT: BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ ; GCN-NEXT: $vgpr10_vgpr11 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ ; GCN-NEXT: S_ENDPGM 0
+ BUFFER_STORE_DWORDX4_OFFEN_exact $vgpr0_vgpr1_vgpr2_vgpr3, $vgpr4, $sgpr8_sgpr9_sgpr10_sgpr11, $sgpr1, 0, 0, 0, implicit $exec
+ $vgpr10_vgpr11 = nofpexcept V_PK_MUL_F32 0, $sgpr0_sgpr1, 8, $vgpr8_vgpr9, 0, 0, 0, 0, 0, implicit $mode, implicit $exec
+ S_ENDPGM 0
+...
>From eb8147239e6d6417064fa789b77aff8f9a83d8a9 Mon Sep 17 00:00:00 2001
From: Sanket Pandit <sanket.pandit at amd.com>
Date: Tue, 19 May 2026 16:59:57 +0000
Subject: [PATCH 2/4] [AMDGPU] Fold gfx940 wide-store WAR hazard scan into
single pass
Address review feedback on #197267:
- Replace the per-window for-loop around getWaitStatesSince with a single
backward scan whose predicate never terminates early and accumulates
the max remaining wait states across every matching producer, using
each producer's own WindowFor() value. Semantically equivalent, but
one CFG walk instead of two, and matches the idiom used by the rest
of GCNHazardRecognizer.cpp.
- Tighten the createsVALUHazard comment to describe the current rule
rather than its history (jayfoad nit).
---
.../lib/Target/AMDGPU/GCNHazardRecognizer.cpp | 36 +++++++++++--------
1 file changed, 21 insertions(+), 15 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
index ad3b40fee440a..c1d73bc7baafc 100644
--- a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
@@ -916,8 +916,8 @@ int GCNHazardRecognizer::createsVALUHazard(const MachineInstr &MI) const {
if (AMDGPU::getRegBitWidth(VDataRCID) > 64) {
// On gfx940-family the BUFFER_STORE source-vgpr WAR hazard exists for
// every SOFFSET shape; the wait-state count differs by SOFFSET, and is
- // computed in checkVALUHazardsHelper. Pre-gfx940 keeps the original
- // narrow gate.
+ // computed in checkVALUHazardsHelper. Pre-gfx940 the hazard only exists
+ // if soffset is not an SGPR.
if (ST.hasGFX940Insts())
return VDataIdx;
const MachineOperand *SOffset =
@@ -972,6 +972,9 @@ int GCNHazardRecognizer::checkVALUHazardsHelper(
const int MaxWaitStates = ST.hasGFX940Insts() ? 2 : 1;
+ // Per-producer required wait-state window. On pre-gfx940 every producer
+ // uses 1; on gfx940-family MUBUF/MTBUF stores with an SGPR SOFFSET use 1
+ // and everything else (literal/absent SOFFSET, FLAT) uses 2.
auto WindowFor = [this, TII](const MachineInstr &MI) -> int {
if (!ST.hasGFX940Insts())
return 1;
@@ -984,19 +987,22 @@ int GCNHazardRecognizer::checkVALUHazardsHelper(
return 2;
};
- for (int Window = 1; Window <= MaxWaitStates; ++Window) {
- auto IsHazardFn = [this, Reg, TRI, Window,
- &WindowFor](const MachineInstr &MI) {
- int DataIdx = createsVALUHazard(MI);
- if (DataIdx < 0)
- return false;
- if (WindowFor(MI) != Window)
- return false;
- return TRI->regsOverlap(MI.getOperand(DataIdx).getReg(), Reg);
- };
- int Elapsed = getWaitStatesSince(IsHazardFn, Window);
- WaitStatesNeeded = std::max(WaitStatesNeeded, Window - Elapsed);
- }
+ // For each hazard producer reached, accumulate the wait states still
+ // needed using that producer's own window. The predicate always returns
+ // false so the walk runs to MaxWaitStates.
+ int Distance = 0;
+ auto Counter = [&](const MachineInstr &MI) {
+ int DataIdx = createsVALUHazard(MI);
+ if (DataIdx >= 0 &&
+ TRI->regsOverlap(MI.getOperand(DataIdx).getReg(), Reg)) {
+ int Need = WindowFor(MI) - Distance;
+ if (Need > WaitStatesNeeded)
+ WaitStatesNeeded = Need;
+ }
+ Distance += SIInstrInfo::getNumWaitStates(MI);
+ return false;
+ };
+ getWaitStatesSince(Counter, MaxWaitStates);
return WaitStatesNeeded;
}
>From 382c18a493e60fc571b7457c292e123bc42877cc Mon Sep 17 00:00:00 2001
From: Sanket Pandit <sanket.pandit at amd.com>
Date: Sun, 31 May 2026 22:14:02 +0000
Subject: [PATCH 3/4] addressing comments
Signed-off-by: Sanket Pandit <sanket.pandit at amd.com>
---
llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
index c1d73bc7baafc..45bdaddb4d0a1 100644
--- a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
@@ -996,8 +996,7 @@ int GCNHazardRecognizer::checkVALUHazardsHelper(
if (DataIdx >= 0 &&
TRI->regsOverlap(MI.getOperand(DataIdx).getReg(), Reg)) {
int Need = WindowFor(MI) - Distance;
- if (Need > WaitStatesNeeded)
- WaitStatesNeeded = Need;
+ WaitStatesNeeded = std::max(WaitStatesNeeded, Need);
}
Distance += SIInstrInfo::getNumWaitStates(MI);
return false;
>From 2febed4f2ba65c81c37157d3b663f3ec4f5cc762 Mon Sep 17 00:00:00 2001
From: Sanket Pandit <sanket.pandit at amd.com>
Date: Mon, 1 Jun 2026 02:46:00 +0000
Subject: [PATCH 4/4] fix distance counting for inline asm
Signed-off-by: Sanket Pandit <sanket.pandit at amd.com>
---
llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
index 45bdaddb4d0a1..aba0f7ff95057 100644
--- a/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNHazardRecognizer.cpp
@@ -998,7 +998,10 @@ int GCNHazardRecognizer::checkVALUHazardsHelper(
int Need = WindowFor(MI) - Distance;
WaitStatesNeeded = std::max(WaitStatesNeeded, Need);
}
- Distance += SIInstrInfo::getNumWaitStates(MI);
+ // Mirror getWaitStatesSince's accounting, which does not count inline asm
+ // towards the wait-state distance.
+ if (!MI.isInlineAsm())
+ Distance += SIInstrInfo::getNumWaitStates(MI);
return false;
};
getWaitStatesSince(Counter, MaxWaitStates);
More information about the llvm-commits
mailing list