[llvm] [AMDGPU] Simplify and document waitcnt handling on call and return (PR #172453)
Jay Foad via llvm-commits
llvm-commits at lists.llvm.org
Tue Dec 16 06:45:43 PST 2025
https://github.com/jayfoad updated https://github.com/llvm/llvm-project/pull/172453
>From 1b128657c8f7d8f280597ca470eff3aae8dc3db0 Mon Sep 17 00:00:00 2001
From: Jay Foad <jay.foad at amd.com>
Date: Tue, 16 Dec 2025 10:26:36 +0000
Subject: [PATCH 1/2] [AMDGPU] Simplify and document waitcnt handling on call
and return
Start documenting the ABI conventions for dependency counters on
function call and return.
Stop pretending that SIInsertWaitcnts can handle anything other than the
default documented behavior.
---
llvm/docs/AMDGPUUsage.rst | 11 +++++---
llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp | 30 ++++-----------------
2 files changed, 13 insertions(+), 28 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 7ecf1c1124894..0554734fd6721 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2066,9 +2066,6 @@ The AMDGPU backend supports the following calling conventions:
stack pointer. Calls to ``amdgpu_gfx`` functions are allowed and behave like they
do in ``amdgpu_cs`` functions.
- All counters (``lgkmcnt``, ``vmcnt``, ``storecnt``, etc.) are presumed in an
- unknown state at function entry.
-
A function may have multiple exits (e.g. one chain exit and one plain ``ret void``
for when the wave ends), but all ``llvm.amdgcn.cs.chain`` exits must be in
uniform control flow.
@@ -2128,6 +2125,14 @@ The AMDGPU backend supports the following calling conventions:
=============================== ==========================================================
+The following ABI conventions apply to all calling conventions that are used for
+callable functions (i.e. those that do not correspond to hardware entry points):
+
+* On entry to a function the dependency counters (``VMcnt``, ``LOADcnt`` etc.)
+ are in an indeterminate state.
+* On return from a function, all dependency counters must be zero except for
+ ``VScnt``/``STOREcnt``.
+
AMDGPU MCExpr
-------------
diff --git a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
index 0a743dbc2e258..18a183bbb4036 100644
--- a/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInsertWaitcnts.cpp
@@ -1893,19 +1893,6 @@ bool WaitcntGeneratorGFX12Plus::createNewWaitcnt(
return Modified;
}
-/// \returns true if the callee inserts an s_waitcnt 0 on function entry.
-static bool callWaitsOnFunctionEntry(const MachineInstr &MI) {
- // Currently all conventions wait, but this may not always be the case.
- //
- // TODO: If IPRA is enabled, and the callee is isSafeForNoCSROpt, it may make
- // senses to omit the wait and do it in the caller.
- return true;
-}
-
-/// \returns true if the callee is expected to wait for any outstanding waits
-/// before returning.
-static bool callWaitsOnFunctionReturn(const MachineInstr &MI) { return true; }
-
/// Generate s_waitcnt instruction to be placed before cur_Inst.
/// Instructions of a given type are returned in order,
/// but instructions of different types can complete out of order.
@@ -1944,8 +1931,7 @@ bool SIInsertWaitcnts::generateWaitcntInstBefore(MachineInstr &MI,
// with knowledge of the called routines.
if (Opc == AMDGPU::SI_RETURN_TO_EPILOG || Opc == AMDGPU::SI_RETURN ||
Opc == AMDGPU::SI_WHOLE_WAVE_FUNC_RETURN ||
- Opc == AMDGPU::S_SETPC_B64_return ||
- (MI.isReturn() && MI.isCall() && !callWaitsOnFunctionEntry(MI))) {
+ Opc == AMDGPU::S_SETPC_B64_return) {
Wait = Wait.combined(WCG->getAllZeroWaitcnt(/*IncludeVSCnt=*/false));
}
// In dynamic VGPR mode, we want to release the VGPRs before the wave exits.
@@ -1993,7 +1979,7 @@ bool SIInsertWaitcnts::generateWaitcntInstBefore(MachineInstr &MI,
if (TII->isAlwaysGDS(Opc) && ScoreBrackets.hasPendingGDS())
addWait(Wait, DS_CNT, ScoreBrackets.getPendingGDSWait());
- if (MI.isCall() && callWaitsOnFunctionEntry(MI)) {
+ if (MI.isCall()) {
// The function is going to insert a wait on everything in its prolog.
// This still needs to be careful if the call target is a load (e.g. a GOT
// load). We also need to check WAW dependency with saved PC.
@@ -2354,15 +2340,9 @@ void SIInsertWaitcnts::updateEventWaitcntAfter(MachineInstr &Inst,
IsSMEMAccess = true;
ScoreBrackets->updateByEvent(SMEM_ACCESS, Inst);
} else if (Inst.isCall()) {
- if (callWaitsOnFunctionReturn(Inst)) {
- // Act as a wait on everything
- ScoreBrackets->applyWaitcnt(
- WCG->getAllZeroWaitcnt(/*IncludeVSCnt=*/false));
- ScoreBrackets->setStateOnFunctionEntryOrReturn();
- } else {
- // May need to way wait for anything.
- ScoreBrackets->applyWaitcnt(AMDGPU::Waitcnt());
- }
+ // Act as a wait on everything
+ ScoreBrackets->applyWaitcnt(WCG->getAllZeroWaitcnt(/*IncludeVSCnt=*/false));
+ ScoreBrackets->setStateOnFunctionEntryOrReturn();
} else if (SIInstrInfo::isLDSDIR(Inst)) {
ScoreBrackets->updateByEvent(EXP_LDS_ACCESS, Inst);
} else if (TII->isVINTERP(Inst)) {
>From 74437d389c39506fcc5b8490f2a3f8aefdcaf78d Mon Sep 17 00:00:00 2001
From: Jay Foad <jay.foad at amd.com>
Date: Tue, 16 Dec 2025 14:45:26 +0000
Subject: [PATCH 2/2] Also document hardware entry points
---
llvm/docs/AMDGPUUsage.rst | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 0554734fd6721..f997a7ab42279 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2133,6 +2133,14 @@ callable functions (i.e. those that do not correspond to hardware entry points):
* On return from a function, all dependency counters must be zero except for
``VScnt``/``STOREcnt``.
+For entry points, the ABI conventions are dictated by the hardware behavior at
+wave launch and wave termination:
+
+* When a wave is launched the shader can assume that all dependency counters are
+ zero.
+* The shader can leave the dependency counters in any state before terminating
+ the wave (e.g. with ``s_endpgm``).
+
AMDGPU MCExpr
-------------
More information about the llvm-commits
mailing list