[llvm] [AMDGPU] Update gfx1250 documentation. NFC (PR #160457)
via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 24 00:56:56 PDT 2025
llvmbot wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-amdgpu
Author: Stanislav Mekhanoshin (rampitec)
<details>
<summary>Changes</summary>
---
Patch is 146.17 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/160457.diff
1 Files Affected:
- (modified) llvm/docs/AMDGPUUsage.rst (+2110-5)
``````````diff
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index edabdc595a1f0..74b7604fda56d 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -979,11 +979,13 @@ supported for the ``amdgcn`` target.
access is not supported except by flat and scratch instructions in
GFX9-GFX11.
- Code that manipulates the stack values in other lanes of a wavefront,
- such as by ``addrspacecast``-ing stack pointers to generic ones and taking offsets
- that reach other lanes or by explicitly constructing the scratch buffer descriptor,
- triggers undefined behavior when it modifies the scratch values of other lanes.
- The compiler may assume that such modifications do not occur.
+ On targets without "Globally Accessible Scratch" (introduced in GFX125x), code that
+ manipulates the stack values in other lanes of a wavefront, such as by
+ ``addrspacecast``-ing stack pointers to generic ones and taking offsets that reach other
+ lanes or by explicitly constructing the scratch buffer descriptor, triggers undefined
+ behavior when it modifies the scratch values of other lanes. The compiler may assume
+ that such modifications do not occur for such targets.
+
When using code object V5 ``LIBOMPTARGET_STACK_SIZE`` may be used to provide the
private segment size in bytes, for cases where a dynamic stack is used.
@@ -1515,6 +1517,88 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
List AMDGPU intrinsics.
+'``llvm.amdgcn.cooperative.atomic``' Intrinsics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``llvm.amdgcn.cooperative.atomic`` :ref:`family of intrinsics<amdgpu-cooperative-atomic-intrinsics-table>`
+provide atomic load and store operations to a naturally-aligned contiguous memory regions.
+Memory is accessed cooperatively by a collection of convergent threads, with each thread accessing
+a fraction of the contiguous memory region.
+
+ .. TODO::
+
+ The memory model described here is imprecise; see SWDEV-536264.
+
+This intrinsic has a memory ordering and may be used to synchronize-with another cooperative atomic.
+If the memory ordering is relaxed, it may pair with a fence if that same fence is executed by
+all participating threads with the same synchronization scope and set of address spaces.
+
+In both cases, a synchronize-with relation can only be established between cooperative atomics with the
+same total access size.
+
+Each target may have additional restrictions on how the intrinsic may be used; see
+:ref:`the table below<amdgpu-llvm-ir-cooperative-atomic-intrinsics-availability>`.
+Targets not covered in the table do not support these intrinsics.
+
+ .. table:: AMDGPU Cooperative Atomic Intrinsics Availability
+ :name: amdgpu-llvm-ir-cooperative-atomic-intrinsics-availability
+
+ =============== =============================================================
+ GFX Version Target Restrictions
+ =============== =============================================================
+ GFX 12.5 :ref:`amdgpu-amdhsa-memory-model-gfx125x-cooperative-atomics`
+ =============== =============================================================
+
+If the intrinsic is used without meeting all of the above conditions, or the target-specific conditions,
+then this intrinsic causes undefined behavior.
+
+ .. table:: AMDGPU Cooperative Atomic Intrinsics
+ :name: amdgpu-cooperative-atomic-intrinsics-table
+
+ ======================================================= =========== ============ ==========
+ LLVM Intrinsic Number of Access Size Total Size
+ Threads Per Thread
+ Used
+ ======================================================= =========== ============ ==========
+ ``llvm.amdgcn.cooperative.atomic.store.32x4B`` 32 4B 128B
+
+ ``llvm.amdgcn.cooperative.atomic.load.32x4B`` 32 4B 128B
+
+ ``llvm.amdgcn.cooperative.atomic.store.16x8B`` 16 8B 128B
+
+ ``llvm.amdgcn.cooperative.atomic.load.16x8B`` 16 8B 128B
+
+ ``llvm.amdgcn.cooperative.atomic.store.8x16B`` 8 16B 128B
+
+ ``llvm.amdgcn.cooperative.atomic.load.8x16B`` 8 16B 128B
+
+ ======================================================= =========== ============ ==========
+
+The intrinsics are available for the global (``.p1`` suffix) and generic (``.p0`` suffix) address spaces.
+
+The atomic ordering operand (3rd operand for ``.store``, 2nd for ``.load``) is an integer that follows the
+C ABI encoding of atomic memory orderings. The supported values are in
+:ref:`the table below<amdgpu-cooperative-atomic-intrinsics-atomic-memory-orderings-table>`.
+
+ .. table:: AMDGPU Cooperative Atomic Intrinsics Atomic Memory Orderings
+ :name: amdgpu-cooperative-atomic-intrinsics-atomic-memory-orderings-table
+
+ ====== ================ =================================
+ Value Atomic Memory Notes
+ Ordering
+ ====== ================ =================================
+ ``0`` ``relaxed`` The default for unsupported values.
+
+ ``2`` ``acquire`` Only for ``.load``
+
+ ``3`` ``release`` Only for ``.store``
+
+ ``5`` ``seq_cst``
+ ====== ================ =================================
+
+The last argument of the intrinsic is the synchronization scope
+as a metadata string, which must be one of the supported :ref:`memory scopes<amdgpu-memory-scopes>`.
+
.. _amdgpu_metadata:
LLVM IR Metadata
@@ -1843,6 +1927,7 @@ The AMDGPU backend supports the following LLVM IR attributes.
This is only relevant on targets with cluster support.
+
================================================ ==========================================================
Calling Conventions
@@ -5261,6 +5346,9 @@ The fields used by CP for code objects before V3 also match those specified in
GFX10-GFX12 (wavefront size 32)
- max_vgpr 1..256
- max(0, ceil(vgprs_used / 8) - 1)
+ GFX125X (wavefront size 32)
+ - max_vgpr 1..1024
+ - max(0, ceil(vgprs_used / 16) - 1)
Where vgprs_used is defined
as the highest VGPR number
@@ -6491,6 +6579,7 @@ following sections:
* :ref:`amdgpu-amdhsa-memory-model-gfx942`
* :ref:`amdgpu-amdhsa-memory-model-gfx10-gfx11`
* :ref:`amdgpu-amdhsa-memory-model-gfx12`
+* :ref:`amdgpu-amdhsa-memory-model-gfx125x`
.. _amdgpu-fence-as:
@@ -16617,6 +16706,2022 @@ the instruction in the code sequence that references the table.
- system for OpenCL.*
============ ============ ============== ========== ================================
+.. _amdgpu-amdhsa-memory-model-gfx125x:
+
+Memory Model GFX125x
+++++++++++++++++++++++++
+
+For GFX125x:
+
+**Device Structure:**
+
+* Each agent has multiple shader engines (SE).
+* Each SE has multiple shader arrays (SA).
+* Each SA has multiple work-group processors (WGP).
+* Each WGP has 4 SIMD32 (2 SIMD32-pairs) that execute wavefronts.
+* The wavefronts for a single work-group are executed in the same
+ WGP.
+
+**Device Memory:**
+
+* Each WGP has a single write-through WGP cache (WGP$) shared by the wavefronts of the
+ work-groups executing on it. The WGP$ is divided between LDS and vector L0 memory.
+
+ * Vector L0 memory holds clean data only.
+
+* Each WGP$ has two request queues; one per SIMD32-pair.
+ Each queue can handle both LDS and vector L0 requests. Requests in one queue
+ are executed serially and in-order, but are not kept in order with the other queue.
+* The scalar memory operations access a scalar L0 cache shared by all wavefronts
+ on a WGP. The scalar and vector L0 caches are not kept coherent by hardware. However, scalar
+ operations are used in a restricted way so do not impact the memory model. See
+ :ref:`amdgpu-amdhsa-memory-spaces`.
+* The vector and scalar memory L0 caches are both clients of an L1 buffer shared by
+ all WGPs on the same SE.
+* L1 buffers have separate request queues for each WGP$ it serves. Requests in one queue
+ are executed serially and in-order, but are not kept in order with other queues.
+* L1 buffers are clients of the L2 cache.
+* There may be multiple L2 caches per agent. Ranges of virtual addresses can be set up as follows:
+
+ * Be non-hardware-coherent; copies of the data are not coherent between multiple L2s.
+ * Be read-write hardware-coherent with other L2 caches on the same or other agents.
+ * Bypass L2 entirely to ensure system coherence.
+
+* L2 caches have multiple memory channels to service disjoint ranges of virtual
+ addresses.
+
+**Memory Model:**
+
+.. note::
+
+ This section is currently incomplete as work on the compiler is still ongoing.
+ The following is a non-exhaustive list of unimplemented/undocumented features:
+ non-volatile bit code sequences, monitor and wait, globally accessing scratch atomics,
+ multicast loads, barriers (including split barriers) and cooperative atomics.
+ Scalar operations memory model needs more elaboration as well.
+
+* Vector memory operations are performed as wavefront wide operations, with the
+ ``EXEC`` mask predicating which lanes execute.
+* Consecutive vector memory operations from the same wavefront are issued in program order.
+ Vector memory operations are issued (and executed) in no particular order between wavefronts.
+* Wave execution of a vector memory operation instruction issues (initiates) the operation,
+ but completion occurs an unspecified amount of time later.
+ The ``s_wait_*cnt`` instructions must be used to determine if the operation has completed.
+* The types of vector memory operations (and their associated ``s_wait_*cnt`` instructions) are:
+
+ * Load (global, scratch, flat, buffer): ``s_wait_loadcnt``
+ * Store (global, scratch, flat, buffer): ``s_wait_storecnt``
+ * non-ASYNC LDS: ``s_wait_dscnt``
+ * ASYNC LDS: ``s_wait_asynccnt``
+ * Tensor: ``s_wait_tensorcnt``
+
+* ``s_wait_xcnt`` is a counter that is incremented when a memory operation is issued, and
+ decremented when memory address translation for that instruction is completed.
+ Waiting on a memory counter ``s_wait_*cnt N`` also waits on ``s_wait_xcnt N``.
+
+ * ``s_wait_xcnt 0x0`` is required before flat and global atomic stores/read-modify-write
+ operations to guarantee atomicity during a xnack replay.
+
+* Within a wavefront, vector memory operation completion (``s_wait_*cnt`` decrement) is
+ reported in order of issue within a type, but in no particular order between types.
+* Within a wavefront, the order in which data is returned to registers by a vector memory
+ operation can be different from the order in which the vector memory operations were issued.
+
+ * Thus, a ``s_wait_*cnt`` instruction must be used to prevent multiple vector memory operations
+ that return results to the same register from executing concurrently as they may not return
+ their results in instruction issue order, even though they will be reported as completed in
+ instruction issue order by the decrementing of the counter.
+
+* Within a wavefront, consecutive loads and store to the same address will be processed in program order
+ by the memory subsystem. Loads and stores to different addresses may be processed
+ out of order with respect to a different address.
+* All non-ASYNC LDS vector memory operations of a WGP are performed as wavefront wide
+ operations in a global order and involve no caching. Completion is reported to a wavefront in
+ execution order.
+* ASYNC LDS and tensor vector memory operations are not covered by the memory model implemented
+ by the AMDGPU backend. Neither ``s_wait_asynccnt`` nor ``s_wait_tensorcnt`` are inserted
+ automatically. They must be emitted using compiler built-in calls.
+* Some vector memory operations contain a ``SCOPE`` field with values
+ corresponding to each cache level. The ``SCOPE`` determines whether a cache
+ can complete an operation locally or whether it needs to forward the operation
+ to the next cache level. The ``SCOPE`` values are:
+
+ * ``SCOPE_CU``: WGP
+ * ``SCOPE_SE``: Shader Engine
+ * ``SCOPE_DEV``: Device/Agent
+ * ``SCOPE_SYS``: System
+
+* Each cache is assigned a ``SCOPE`` by the hardware depending on the agent's
+ configuration.
+
+ * This ensures that ``SCOPE_DEV`` can always be used to implement agent coherence,
+ even in the presence of multiple non-coherent L2 caches on the same agent.
+
+* When a vector memory operation with a given ``SCOPE`` reaches a cache with a smaller
+ ``SCOPE`` value, it is forwarded to the next level of cache.
+* When a vector memory operation with a given ``SCOPE`` reaches a cache with a ``SCOPE``
+ value greater than or equal to its own, the operation can proceed:
+
+ * Reads can hit into the cache.
+ * Writes can happen in this cache and completion (``s_wait`` decrement) can be
+ reported.
+ * RMW operations can be done locally.
+
+* Some memory operations contain a ``nv`` bit, for "non-volatile", which indicates
+ memory that is not expected to change during a kernel's execution.
+ This information is propagated to the cache lines for that address
+ (refered to as ``$nv``).
+
+ * When ``nv=0`` reads hit dirty ``$nv=1`` data in cache, the hardware will
+ writeback the data to the next level in the hierarchy and then subsequently read
+ it again, updating the cache line with a clean ``$nv=0`` copy of the data.
+
+* ``global_inv``, ``global_wb`` and ``global_wbinv`` are cache control instructions.
+ The affected cache(s) are controlled by the ``SCOPE`` of the instruction.
+ Only caches whose scope is strictly smaller than the instruction's are affected.
+
+ * ``global_inv`` invalidates the data in affected caches so that subsequent reads
+ will re-read from the next level in the cache hierarchy.
+ The invalidation requests cannot be reordered with pending or upcoming
+ memory operations. Instruction completion is reported using ``s_wait_loadcnt``.
+ * ``global_wb`` flushes the dirty data in affected caches to the next level in
+ the cache hierarchy. This instruction additionally ensures previous
+ memory operation done at a lower scope level have reached the desired
+ ``SCOPE:``. Instruction completion is reported using ``s_wait_storecnt`` once
+ all data has been acknowledged by the next level in the cache hierarchy.
+ * ``global_wbinv`` performs a ``global_inv`` then a ``global_wb``.
+ Instruction completion is reported using ``s_wait_storecnt``.
+ * ``global_inv``, ``global_wb`` and ``global_wbinv`` with ``nv=0`` can only
+ affect ``$nv=0`` cache lines, whereas ``nv=1`` can affect all cache lines.
+ * ``global_inv``, ``global_wb`` and ``global_wbinv`` behave like memory operations
+ issued to every address at the same time. They are kept in order with other
+ memory operations from the same wave.
+
+Scalar memory operations are only used to access memory that is proven to not
+change during the execution of the kernel dispatch. This includes constant
+address space and global address space for program scope ``const`` variables.
+Therefore, the kernel machine code does not have to maintain the scalar cache to
+ensure it is coherent with the vector caches. The scalar and vector caches are
+invalidated between kernel dispatches by CP since constant address space data
+may change between kernel dispatch executions. See
+:ref:`amdgpu-amdhsa-memory-spaces`.
+
+Atomics in the scratch address space are handled as follows:
+
+* Data types <= 32 bits: The instruction is converted into an atomic in the
+ generic (``flat``) address space. All properties of the atomic
+ (atomic ordering, volatility, alignment, etc.) are preserved.
+ Refer to the generic address space code sequences for further information.
+* Data types >32 bits: unsupported and an error is emitted.
+
+The code sequences used to implement the memory model for GFX125x are defined in
+table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx125x-table`.
+
+The mapping of LLVM IR syncscope to GFX125x instruction ``scope`` operands is
+defined in :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx125x-scopes-table`.
+
+The table only applies if and only if it is directly referenced by an entry in
+:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx125x-table`, and it only applies to
+the instruction in the code sequence that references the table.
+
+ .. table:: AMDHSA Memory Model Code Sequences GFX125x - Instruction Scopes
+ :name: amdgpu-amdhsa-memory-model-code-sequences-gfx125x-scopes-table
+
+ ================================= =======================
+ LLVM syncscope ISA
+
+
+ ================================= =======================
+ *none*, one-as ``scope:SCOPE_SYS``
+ system, system-one-as ``scope:SCOPE_SYS``
+ agent, agent-one-as ``scope:SCOPE_DEV``
+ cluster, cluster-one-as ``scope:SCOPE_SE``
+ workgroup, workgroup-one-as ``scope:SCOPE_CU`` [1]_
+ wavefront, wavefront-one-as ``scope:SCOPE_CU`` [1]_
+ singlethread, singlethread-one-as ``scope:SCOPE_CU`` [1]_
+ ================================= =======================
+
+ .. [1] ``SCOPE_CU`` is the default ``scope:`` emitted by the compiler.
+ It will be omitted when instructions are emitted in textual form by the compiler.
+
+ .. table:: AMDHSA Memory Model Code Sequences GFX125x
+ :name: amdgpu-amdhsa-memory-model-code-sequences-gfx125x-table
+
+ ============ ============ ============== ========== ================================
+ LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code
+ Ordering Sync Scope Address GFX125x
+ Space
+ ============ ============ ============== ========== ================================
+ **Non-Atomic**
+ ------------------------------------------------------------------------------------
+ load *none* *none* - global - !volatile & !nontemporal
+ - generic
+ - private 1. buffer/global/flat_load
+ - constant
+ - !volatile & nontemporal
+
+ 1. buffer/global/flat_load
+ ``th:TH_LOAD_NT``
+
+ - volatile
+
+ 1. buffer/global/flat_load
+ ``scope:SCOPE_SYS``
+
+ 2. ``s_wait_loadcnt 0x0``
+
+ - Must happen before
+ any following volatile
+ global/generic
+ load/store.
+ - Ensures that
+ volatile
+ operations to
+ different
+ addresses will not
+ be reordered by
+ hardware.
...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/160457
More information about the llvm-commits
mailing list