[llvm] [docs][AMDGPU] move DMA operations to a separate file (PR #206917)
Sameer Sahasrabuddhe via llvm-commits
llvm-commits at lists.llvm.org
Thu Jul 2 01:23:06 PDT 2026
https://github.com/ssahasra updated https://github.com/llvm/llvm-project/pull/206917
>From b3ec327d969c8ba144c59fa4b4437f43fd20f0da Mon Sep 17 00:00:00 2001
From: Sameer Sahasrabuddhe <sameer.sahasrabuddhe at amd.com>
Date: Wed, 1 Jul 2026 13:56:59 +0530
Subject: [PATCH 1/2] [docs][AMDGPU] move DMA operations to a separate file
---
llvm/docs/AMDGPUAsyncOperations.rst | 56 ++---------
llvm/docs/AMDGPUDMAOperations.md | 145 ++++++++++++++++++++++++++++
llvm/docs/AMDGPUUsage.rst | 54 +++++------
llvm/docs/UserGuides.md | 8 +-
4 files changed, 188 insertions(+), 75 deletions(-)
create mode 100644 llvm/docs/AMDGPUDMAOperations.md
diff --git a/llvm/docs/AMDGPUAsyncOperations.rst b/llvm/docs/AMDGPUAsyncOperations.rst
index 548bbfb86bcf3..93f5cf10ff448 100644
--- a/llvm/docs/AMDGPUAsyncOperations.rst
+++ b/llvm/docs/AMDGPUAsyncOperations.rst
@@ -10,52 +10,14 @@
Introduction
============
-Asynchronous operations are operations that are completed independently at an
-unspecified scope. A thread that initiates one or more async operations can use
+Asynchronous operations are operations whose completion is not tracked
+internally by the compiler. A thread that initiates one or more async operations can use
*asyncmarks* to track their completion.
-Operations
-==========
-
-Async Instructions
-------------------
-
-The following instructions initiate async operations that transfer data between
-global memory and LDS memory.
-
-.. note::
-
- These listings are *merely representative*. The actual function signatures
- and supported architectures are documented in the :ref:`amdgpu-usage-guide`.
-
-**GFX9 Async Instructions (LDS DMA)**
-
-.. code-block:: llvm
-
- void @llvm.amdgcn.load.async.to.lds(ptr %src, ptr %dst)
- void @llvm.amdgcn.global.load.async.lds(ptr %src, ptr %dst)
- void @llvm.amdgcn.raw.buffer.load.async.lds(ptr %src, ptr %dst)
- void @llvm.amdgcn.raw.ptr.buffer.load.async.lds(ptr %src, ptr %dst)
- void @llvm.amdgcn.struct.buffer.load.async.lds(ptr %src, ptr %dst)
- void @llvm.amdgcn.struct.ptr.buffer.load.async.lds(ptr %src, ptr %dst)
-
-**GFX12 Async Instructions**
-
-.. code-block:: llvm
-
- void @llvm.amdgcn.global.load.async.to.lds.type(ptr %dst, ptr %src)
- void @llvm.amdgcn.global.store.async.from.lds.type(ptr %dst, ptr %src)
- void @llvm.amdgcn.cluster.load.async.to.lds.type(ptr %dst, ptr %src)
-
-**GFX1250 Tensor DMA Instructions**
-
-.. code-block:: llvm
-
- void @llvm.amdgcn.tensor.load.to.lds(...)
- void @llvm.amdgcn.tensor.store.from.lds(...)
+- Most :ref:`DMA operations<amdgpu-dma-operations>` are asynchronous.
Asyncmarks
-----------
+==========
An *asyncmark* created by a thread can be used to track async operations
initiated by that thread. The abstract machine maintains a sequence of
@@ -65,18 +27,20 @@ executing function. The state of this sequence at each program point in the
function is called the *current sequence*.
``@llvm.amdgcn.asyncmark()``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+----------------------------
Produces an asyncmark and appends it to the current sequence.
``@llvm.amdgcn.wait.asyncmark(i16 %N)``
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+---------------------------------------
Ensures that the length of the current sequence is at most ``N`` by removing
asyncmarks from the start of the sequence if it is more than ``N``.
-Memory Consistency Model
-========================
+.. _amdgpu-asyncmark-memory-model:
+
+Memory Model
+============
An ``asyncmark()`` operation ``X`` that produces an asyncmark ``M`` is
*completed-at* a ``wait.asyncmark()`` operation ``Y`` in the same function body
diff --git a/llvm/docs/AMDGPUDMAOperations.md b/llvm/docs/AMDGPUDMAOperations.md
new file mode 100644
index 0000000000000..7d122c3d5fb5d
--- /dev/null
+++ b/llvm/docs/AMDGPUDMAOperations.md
@@ -0,0 +1,145 @@
+(amdgpu-dma-operations)=
+
+# AMDGPU DMA Operations
+
+```{contents}
+:local:
+```
+
+## Introduction
+
+DMA operations transfer data between different kinds of memory directly without
+occupying registers in the invoking wave. They are usually
+{ref}`asynchronous<amdgpu-async-operations>` asynchronous, and require the user
+to explicitly track completion using {ref}`asyncmarks<amdgpu-async-operations>`.
+
+All DMA operations support the same cache modifiers as ordinary load/store
+operations from registers. They cannot be performed atomically.
+
+### GFX9 DMA
+
+Each GFX9 DMA instruction has a synchronous counterpart (e.g.,
+``@llvm.amdgcn.load.to.lds`` for ``@llvm.amdgcn.load.async.to.lds``). The
+synchronous variants perform the same operation, but the compiler automatically
+ensures completion before their side-effects are used.
+
+GFX9 DMA instructions implement volatile (via ``aux/cpol`` bit 31) and
+nontemporal (via metadata) as if they were loads from the global address space.
+
+**Flat/Global Addressing**
+
+```llvm
+void @llvm.amdgcn.load[.async].to.lds.pN(
+ ptr addrspace(N) %src, ; base pointer to load from (per-lane)
+ ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform)
+ i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950)
+ i32 immarg %offset, ; offset applied to both src and LDS address
+ i32 immarg %cpol) ; cache policy
+```
+
+Loads data from global memory to LDS. The data size can be 1, 2, or 4 bytes
+(gfx950 also allows 12 or 16 bytes). The LDS address is implicitly offset by
+``4 * lane_id`` bytes for sizes up to 4 bytes, and by ``16 * lane_id`` bytes
+for larger sizes.
+
+The ``%lds_base`` pointer must be wave-uniform.
+
+The source pointer is overloaded on address space. Supported address spaces are
+flat (0), global (1), and buffer fat pointer (7).
+
+``@llvm.amdgcn.load[.async].to.lds.p7`` (buffer pointer) is lowered to
+``@llvm.amdgcn.raw.ptr.buffer.load[.async].lds`` before instruction selection.
+
+```llvm
+void @llvm.amdgcn.global.load[.async].lds(
+ ptr addrspace(1) %src, ; global base pointer to load from (per-lane)
+ ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform)
+ i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950)
+ i32 immarg %offset, ; offset applied to both global and LDS address
+ i32 immarg %cpol) ; cache policy
+```
+
+This is identical to ``@llvm.amdgcn.load[.async].to.lds.p1``.
+
+**Buffer Addressing**
+
+```llvm
+void @llvm.amdgcn.{raw|struct}[.ptr].buffer.load[.async].lds(
+ %rsrc, ; buffer resource descriptor (SGPR):
+ ; <4 x i32> or ptr addrspace(8)
+ ptr addrspace(3) %lds_base, ; LDS base pointer (wave-uniform)
+ i32 immarg %size, ; data byte size: 1/2/4 (12/16 for gfx950)
+ [i32 %vindex,] ; VGPR buffer index (struct variants only)
+ i32 %voffset, ; VGPR offset (included in bounds checking)
+ i32 %soffset, ; SGPR/imm offset (excluded from bounds checking)
+ i32 immarg %offset, ; imm offset (included in bounds checking)
+ i32 immarg %cpol) ; cache policy
+```
+
+Loads data from a buffer resource to LDS.
+
+The ``%lds_base`` pointer must be wave-uniform.
+
+The intrinsics differ in two orthogonal ways:
+
+- **raw** vs **struct**: The ``struct`` variants add a ``%vindex`` argument for
+ indexed buffer addressing.
+- **ptr** vs non-ptr: The ``ptr`` variants use ``ptr addrspace(8)`` for the
+ buffer resource descriptor; the non-ptr variants use ``<4 x i32>``.
+
+### GFX1250
+
+GFX1250 LDS DMA instructions implement nontemporal (via metadata) as if they
+were loads from the global address space. Tensor DMA instructions do not support
+volatile or nontemporal.
+
+**Global Addressing**
+
+```llvm
+void @llvm.amdgcn.{global|cluster}.load.async.to.lds.b<N>(
+ ptr addrspace(1) %src, ; global base pointer to load from (per-lane)
+ ptr addrspace(3) %lds_base, ; LDS base pointer (per-lane)
+ i32 immarg %offset, ; offset applied to both global and LDS address
+ i32 immarg %cpol, ; cache policy
+ [i32 %m0]) ; workgroup broadcast mask, cluster variants only (in M0)
+```
+
+The bit-size encoded in the name can be 8, 32, 64 or 128.
+
+Loads data from global memory to LDS. The ``%offset`` is applied to both the
+global and LDS addresses.
+
+The ``cluster`` variants add a ``%m0`` argument for workgroup broadcast. The
+broadcast mask selects which workgroups within a cluster participate in the load.
+
+```llvm
+void @llvm.amdgcn.global.store.async.from.lds.b<N>(
+ ptr addrspace(1) %dst, ; global base pointer to store to (per-lane)
+ ptr addrspace(3) %lds_base, ; LDS base pointer to load from (per-lane)
+ i32 immarg %offset, ; offset applied to both global and LDS address
+ i32 immarg %cpol) ; cache policy
+```
+
+Stores data from LDS to global memory.
+
+**Tensor Addressing**
+
+```llvm
+void @llvm.amdgcn.tensor.{load.to|store.from}.lds(
+ <4 x i32> %desc0, ; D# group 0
+ <8 x i32> %desc1, ; D# group 1
+ <4 x i32> %desc2, ; D# group 2 (zero-init for D# up to 2D)
+ <4 x i32> %desc3, ; D# group 3 (zero-init for D# up to 2D)
+ <8 x i32> %desc4, ; D# group 4 (reserved, use zeroinitializer)
+ i32 immarg %cpol) ; cache policy
+```
+
+Loads or stores data between global memory and LDS using a tensor descriptor
+(D#). The descriptor is split across multiple groups. GFX1250 supports up to 4
+descriptor groups; ``%desc4`` is reserved for future targets and must be
+zero-initialized.
+
+Despite the absence of ``.async`` in their names, these intrinsics are
+asynchronous.
+
+All arguments must be wave-uniform.
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 2b522395ee892..0129c5e4f8994 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1520,21 +1520,6 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
The format is a 64-bit concatenation of the MODE and TRAPSTS registers.
:ref:`llvm.set.fpenv<int_set_fpenv>` Sets the floating point environment to the specified state.
- llvm.amdgcn.load.to.lds.p<1/7> Loads values from global memory (either in the form of a global
- a raw fat buffer pointer) to LDS. The size of the data copied can be 1, 2,
- or 4 bytes (and gfx950 also allows 12 or 16 bytes). The LDS pointer
- argument should be wavefront-uniform; the global pointer need not be.
- The LDS pointer is implicitly offset by 4 * lane_id bytes for size <= 4 bytes
- and 16 * lane_id bytes for larger sizes. This lowers to `global_load_lds`,
- `buffer_load_* ... lds`, or `global_load__* ... lds` depending on address
- space and architecture. `amdgcn.global.load.lds` has the same semantics as
- `amdgcn.load.to.lds.p1`.
-
- llvm.amdgcn.load.async.to.lds.p<1/7> Same as `llvm.amdgcn.load.to.lds.p<1/7>`, but the completion of this
- :ref:`asynchronous version<amdgpu-async-operations>` is not automatically tracked
- by the compiler. The user must explicitly track the completion with `asyncmark`
- operations before using their side-effects.
-
llvm.amdgcn.readfirstlane Provides direct access to v_readfirstlane_b32. Returns the value in
the lowest active lane of the input operand. Currently implemented
for i16, i32, float, half, bfloat, <2 x i16>, <2 x half>, <2 x bfloat>,
@@ -2097,6 +2082,31 @@ The 3rd operand for ``.store`` or 2nd for ``.load`` intrinsics is the
The last operand of the intrinsic is the
:ref:`synchronization scope<amdgpu-intrinsics-syncscope-metadata-operand>` of the operation.
+DMA Intrinsics
+~~~~~~~~~~~~~~
+
+DMA intrinsics transfer data between global memory and LDS without occupying
+registers. See :ref:`amdgpu-dma-operations` for full documentation.
+
+::
+
+ llvm.amdgcn.load[.async].to.lds
+ llvm.amdgcn.global.load[.async].lds
+ llvm.amdgcn.{raw|struct}[.ptr].buffer.load[.async].lds
+ llvm.amdgcn.{global|cluster}.load.async.to.lds.b{8,32,64,128}
+ llvm.amdgcn.global.store.async.from.lds.b{8,32,64,128}
+
+Tensor Intrinsics
+~~~~~~~~~~~~~~~~~
+
+Tensor intrinsics transfer data between global memory and LDS using a tensor
+descriptor. Despite the absence of ``.async`` in their names, these intrinsics
+are asynchronous. See :ref:`amdgpu-dma-operations` for full documentation.
+
+::
+
+ llvm.amdgcn.tensor.{load.to|store.from}.lds
+
Intrinsic Operands
~~~~~~~~~~~~~~~~~~
@@ -7204,19 +7214,7 @@ operations.
termed vector memory operations.
``global_load_lds`` or ``buffer/global_load`` instructions with the `lds` flag
-are LDS DMA loads. They interact with caches as if the loaded data were
-being loaded to registers and not to LDS, and so therefore support the same
-cache modifiers. They cannot be performed atomically. They implement volatile
-(via aux/cpol bit 31) and nontemporal (via metadata) as if they were loads
-from the global address space.
-
-The LDS DMA instructions are synchronous by default, which means that the
-compiler will automatically ensure that the corresponding operation has
-completed before its side-effects are used. The :ref:`asynchronous
-versions<amdgpu-async-operations>` of these same instructions perform the same
-operations, but without automatic tracking in the compiler; the user must
-explicitly track the completion of these instructions before using their
-side-effects.
+are :ref:`LDS DMA operations<amdgpu-dma-operations>`.
Private address space uses ``buffer_load/store`` using the scratch V#
(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11). Since only a single thread
diff --git a/llvm/docs/UserGuides.md b/llvm/docs/UserGuides.md
index e91cdd8975115..96024beae9fe1 100644
--- a/llvm/docs/UserGuides.md
+++ b/llvm/docs/UserGuides.md
@@ -63,6 +63,7 @@ CompileCudaWithLLVM
NVPTXUsage
AMDGPUUsage
AMDGPUAsyncOperations
+AMDGPUDMAOperations
AMDGPUExecutionSynchronization
AMDGPUMemoryModel
SPIRVUsage
@@ -338,7 +339,12 @@ yaml2obj
- {doc}`AMDGPUAsyncOperations`
- Builtins for invoking asynchronous data transfer operations in AMD GPUs.
+ Asynchronous operations and the asyncmark mechanism for tracking their
+ completion on AMD GPUs.
+
+- {doc}`AMDGPUDMAOperations`
+
+ DMA operations that transfer data between global memory and LDS on AMD GPUs.
- {doc}`AMDGPUMemoryModel`
>From c27d678797835e97b4f36d7a3d3310d842b207bb Mon Sep 17 00:00:00 2001
From: Sameer Sahasrabuddhe <sameer.sahasrabuddhe at amd.com>
Date: Thu, 2 Jul 2026 13:50:56 +0530
Subject: [PATCH 2/2] expand "DMA" at least once
---
llvm/docs/AMDGPUDMAOperations.md | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/llvm/docs/AMDGPUDMAOperations.md b/llvm/docs/AMDGPUDMAOperations.md
index 7d122c3d5fb5d..3de4be7c3e007 100644
--- a/llvm/docs/AMDGPUDMAOperations.md
+++ b/llvm/docs/AMDGPUDMAOperations.md
@@ -8,10 +8,11 @@
## Introduction
-DMA operations transfer data between different kinds of memory directly without
-occupying registers in the invoking wave. They are usually
-{ref}`asynchronous<amdgpu-async-operations>` asynchronous, and require the user
-to explicitly track completion using {ref}`asyncmarks<amdgpu-async-operations>`.
+DMA (or "Direct Memory Access") operations transfer data between different kinds
+of memory directly without occupying registers in the invoking wave. They are
+usually {ref}`asynchronous<amdgpu-async-operations>` asynchronous, and require
+the user to explicitly track completion using
+{ref}`asyncmarks<amdgpu-async-operations>`.
All DMA operations support the same cache modifiers as ordinary load/store
operations from registers. They cannot be performed atomically.
More information about the llvm-commits
mailing list