[llvm] [AMDGPU] Improve the description of asyncmark semantics (PR #202579)
Sameer Sahasrabuddhe via llvm-commits
llvm-commits at lists.llvm.org
Thu Jun 11 23:30:57 PDT 2026
https://github.com/ssahasra updated https://github.com/llvm/llvm-project/pull/202579
>From f744f486be194d29659314dc150d327535610033 Mon Sep 17 00:00:00 2001
From: Sameer Sahasrabuddhe <sameer.sahasrabuddhe at amd.com>
Date: Tue, 9 Jun 2026 16:33:16 +0530
Subject: [PATCH 1/3] [AMDGPU] Improve the description of asyncmark semantics
- The semantics of asyncmarks is now definded purely in terms of sequences,
without referring to the implementation.
- The examples incorrectly used (post)dominance. Fixed that with wording in
terms of asyncmark sequences.
---
llvm/docs/AMDGPUAsyncOperations.rst | 164 +++++++++++++++++++---------
1 file changed, 113 insertions(+), 51 deletions(-)
diff --git a/llvm/docs/AMDGPUAsyncOperations.rst b/llvm/docs/AMDGPUAsyncOperations.rst
index 0b8ea0ae77174..90aac42088be2 100644
--- a/llvm/docs/AMDGPUAsyncOperations.rst
+++ b/llvm/docs/AMDGPUAsyncOperations.rst
@@ -13,9 +13,7 @@ Introduction
Asynchronous operations are memory transfers (usually between the global memory
and LDS) that are completed independently at an unspecified scope. A thread that
requests one or more asynchronous transfers can use *asyncmarks* to track
-their completion. The thread waits for each asyncmark to be *completed*, which
-indicates that requests initiated in *program-order* before this asyncmark have also
-completed.
+their completion.
Operations
==========
@@ -57,49 +55,53 @@ memory and LDS memory.
void @llvm.amdgcn.tensor.load.to.lds(...)
void @llvm.amdgcn.tensor.store.from.lds(...)
-Asyncmark Operations
----------------------
+Asyncmarks
+----------
An *asyncmark* in the abstract machine tracks all the async operations that
-are *program-ordered* before that asyncmark. An asyncmark M is said to be *completed*
-only when all async operations *program-ordered* before M are reported by the
-implementation as having finished, and it is said to be *outstanding* otherwise.
-
-Thus we have the following sufficient condition:
-
- An async operation X is *completed* at a program point P if there exists an
- asyncmark M such that X is *program-ordered* before M, M is *program-ordered* before
- P, and M is completed. X is said to be *outstanding* at P otherwise.
+are *program-ordered* before that asyncmark.
The abstract machine maintains a sequence of asyncmarks during the
execution of a function body, which excludes any asyncmarks produced by calls to
other functions encountered in the currently executing function.
+This sequence is called the *current sequence* of that function body.
``@llvm.amdgcn.asyncmark()``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-When executed, inserts an asyncmark in the sequence associated with the
-currently executing function body.
+Appends an asyncmark to the current sequence.
``@llvm.amdgcn.wait.asyncmark(i16 %N)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Waits until there are at most N outstanding asyncmarks in the sequence associated
-with the currently executing function body.
+Ensures that the length of the current sequence is at most ``N`` by removing
+asyncmarks from the start of the sequence if it is more than ``N``.
Memory Consistency Model
========================
-Each asynchronous operation consists of a non-atomic read on the source and a
-non-atomic write on the destination. Async "LDS DMA" intrinsics result in async
-accesses that guarantee visibility relative to other memory operations as
-follows:
+An ``asyncmark()`` operation ``X`` that inserts an asyncmark ``M`` is
+*completed-at* a ``wait.asyncmark()`` operation ``Y`` in the same function body
+if:
+
+- ``X`` is *program-ordered* before ``Y``, and
+- ``M`` is not in the current sequence after ``Y`` returns.
+
+An async operation ``A`` is *completed-at* a ``wait.asyncmark()`` operation
+``Y`` if there exists an ``asyncmark()`` operation ``X`` such that:
+
+- ``A`` is *program-ordered* before ``X``, and
+- ``X`` is *completed-at* ``Y``.
- An asynchronous operation `A` program ordered before an overlapping memory
- operation `X` happens-before `X` only if `A` is completed before `X`.
+An asynchronous operation ``A`` *happens-before* an overlapping memory operation
+``B`` only if there exists a ``wait.asyncmark()`` operation ``Y`` such that:
- A memory operation `X` program ordered before an overlapping asynchronous
- operation `A` happens-before `A`.
+- ``A`` is *program-ordered* before ``Y``, and
+- ``Y`` is *program-ordered* before ``B``, and
+- ``A`` is *completed-at* ``Y``.
+
+A memory operation ``B`` *happens-before* an overlapping asynchronous
+operation ``A`` if ``B`` is *program-ordered* before ``A``.
.. note::
@@ -192,8 +194,11 @@ Ordinary function call
// third block
asyncmark();
- wait.asyncmark(1); // wait for the second block
- wait.asyncmark(0); // will wait for third block, including bar()
+ // wait for the second block
+ wait.asyncmark(1);
+
+ // wait for the third block, including bar()
+ wait.asyncmark(0);
}
Implementation notes
@@ -201,17 +206,8 @@ Implementation notes
[This section is informational.]
-Optimization
-------------
-
-The implementation may eliminate asyncmark/wait intrinsics in the following cases:
-
-1. An ``asyncmark`` operation which is not included in the wait count of a later
- wait operation in the current function. In particular, an ``asyncmark`` which
- is not post-dominated by any ``wait.asyncmark``.
-2. A ``wait.asyncmark`` whose wait count is more than the outstanding async
- asyncmarks at that point. In particular, a ``wait.asyncmark`` that is not
- dominated by any ``asyncmark``.
+Function Calls
+--------------
In general, at a function call, if the caller uses sufficient waits to track
its own async operations, the actions performed by the callee cannot affect
@@ -220,30 +216,96 @@ correctness. But inlining such a call may result in redundant waits.
.. code-block:: c++
void foo() {
- asyncmark(); // A
+ ...
+ asyncmark(); // X
+ ... // no wait.asyncmark()
}
void bar() {
- asyncmark(); // B
- asyncmark(); // C
+ asyncmark(); // B
+ asyncmark(); // C
foo();
- wait.asyncmark(1);
+ wait.asyncmark(1); // D
+ }
+
+Before inlining, it is unspecified whether ``X`` is *completed-at* ``D``, while
+``C`` is **not** *completed-at* ``D``. The programmer can only rely on ``B``
+being *completed-at* ``D``.
+
+.. code-block:: c++
+
+ void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ asyncmark(); // X
+ wait.asyncmark(1); // D
}
-Before inlining, the ``wait.asyncmark`` waits for asyncmark B to be completed.
+After inlining, ``C`` is also *completed-at* ``D`` and ``X`` is **not**
+*completed-at* ``D``.
+
+Conversely, a ``wait.asyncmark`` call inside a callee cannot be used to track
+asyncmarks inserted by the caller, since this ``wait.asyncmark`` can only
+observe the current sequence of the callee.
.. code-block:: c++
void foo() {
+ ... // no asyncmark()
+ wait.asyncmark(0); // Y
+ ...
}
void bar() {
- asyncmark(); // B
- asyncmark(); // C
- asyncmark(); // A from call to foo()
- wait.asyncmark(1);
+ asyncmark(); // B
+ asyncmark(); // C
+ foo();
+ wait.asyncmark(1); // D
+ }
+
+In the above example, it is unspecified whether ``B`` and ``C`` in ``bar()`` are
+*completed-at* ``Y``, because they are not included in the sequence that can be
+examined at ``Y``.
+
+.. code-block:: c++
+
+ void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ wait.asyncmark(0); // Y
+ wait.asyncmark(1); // D
}
-After inlining, the ``wait.asyncmark`` now waits for asyncmark C to complete, which is
-longer than necessary. Ideally, the optimizer should have eliminated asyncmark A in
-the body of foo() itself.
+After inlining, both ``B`` and ``C`` are *completed-at* ``Y``.
+
+Optimization
+------------
+
+The implementation may eliminate asyncmark/wait intrinsics in the following
+cases. These are just examples and not meant to be an exhaustive list.
+
+1. An ``asyncmark`` operation which remains in the current sequence along every
+ path that reaches the function exit.
+
+ .. code-block:: c++
+
+ void foo() {
+ ...
+ asyncmark(); // X
+ ... // no wait.asyncmark()
+ }
+
+ Here, ``X`` can be eliminated.
+
+2. A ``wait.asyncmark`` which sees an empty sequence of asyncmarks along every
+ path that reaches it.
+
+ .. code-block:: c++
+
+ void foo() {
+ ... // no asyncmark()
+ wait.asyncmark(0); // Y
+ ...
+ }
+
+ Here, ``Y`` can be eliminated.
>From 69d4f00fcaa1970e4d2c65481d752538f7531126 Mon Sep 17 00:00:00 2001
From: Sameer Sahasrabuddhe <sameer.sahasrabuddhe at amd.com>
Date: Thu, 11 Jun 2026 10:37:53 +0530
Subject: [PATCH 2/3] tighten the defintion of *completed-at*
---
llvm/docs/AMDGPUAsyncOperations.rst | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/llvm/docs/AMDGPUAsyncOperations.rst b/llvm/docs/AMDGPUAsyncOperations.rst
index 90aac42088be2..0f66df107970b 100644
--- a/llvm/docs/AMDGPUAsyncOperations.rst
+++ b/llvm/docs/AMDGPUAsyncOperations.rst
@@ -85,7 +85,8 @@ An ``asyncmark()`` operation ``X`` that inserts an asyncmark ``M`` is
if:
- ``X`` is *program-ordered* before ``Y``, and
-- ``M`` is not in the current sequence after ``Y`` returns.
+- ``M`` is not in the current sequence at any operation ``Z`` that immediately
+ follows ``Y`` in *program-order*.
An async operation ``A`` is *completed-at* a ``wait.asyncmark()`` operation
``Y`` if there exists an ``asyncmark()`` operation ``X`` such that:
>From e9f3104474119feb49ba8ef5914197e255cf4fe6 Mon Sep 17 00:00:00 2001
From: Sameer Sahasrabuddhe <sameer.sahasrabuddhe at amd.com>
Date: Fri, 12 Jun 2026 11:59:04 +0530
Subject: [PATCH 3/3] replace po with hb; clarify DMA v/s "async"; inline "..."
---
llvm/docs/AMDGPUAsyncOperations.rst | 59 ++++++++++++-----------------
1 file changed, 25 insertions(+), 34 deletions(-)
diff --git a/llvm/docs/AMDGPUAsyncOperations.rst b/llvm/docs/AMDGPUAsyncOperations.rst
index 0f66df107970b..08f5206d5a6a3 100644
--- a/llvm/docs/AMDGPUAsyncOperations.rst
+++ b/llvm/docs/AMDGPUAsyncOperations.rst
@@ -10,19 +10,18 @@
Introduction
============
-Asynchronous operations are memory transfers (usually between the global memory
-and LDS) that are completed independently at an unspecified scope. A thread that
-requests one or more asynchronous transfers can use *asyncmarks* to track
-their completion.
+Asynchronous operations are operations that are completed independently at an
+unspecified scope. A thread that requests one or more async operations can use
+*asyncmarks* to track their completion.
Operations
==========
-Memory Accesses
----------------
+Async Instructions
+------------------
-The following instructions request asynchronous transfer of data between global
-memory and LDS memory.
+The following instructions request async operations that transfer data between
+global memory and LDS memory.
.. note::
@@ -58,13 +57,12 @@ memory and LDS memory.
Asyncmarks
----------
-An *asyncmark* in the abstract machine tracks all the async operations that
-are *program-ordered* before that asyncmark.
-
-The abstract machine maintains a sequence of asyncmarks during the
-execution of a function body, which excludes any asyncmarks produced by calls to
-other functions encountered in the currently executing function.
-This sequence is called the *current sequence* of that function body.
+An *asyncmark* created by a thread can be used to track async operations
+initiated by that thread. The abstract machine maintains a sequence of
+asyncmarks during the execution of a function body, which excludes any
+asyncmarks produced by calls to other functions encountered in the currently
+executing function. The state of this sequence at each program point in the
+function is called the *current sequence*.
``@llvm.amdgcn.asyncmark()``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -88,27 +86,16 @@ if:
- ``M`` is not in the current sequence at any operation ``Z`` that immediately
follows ``Y`` in *program-order*.
-An async operation ``A`` is *completed-at* a ``wait.asyncmark()`` operation
-``Y`` if there exists an ``asyncmark()`` operation ``X`` such that:
-
-- ``A`` is *program-ordered* before ``X``, and
-- ``X`` is *completed-at* ``Y``.
-
-An asynchronous operation ``A`` *happens-before* an overlapping memory operation
-``B`` only if there exists a ``wait.asyncmark()`` operation ``Y`` such that:
-
-- ``A`` is *program-ordered* before ``Y``, and
-- ``Y`` is *program-ordered* before ``B``, and
-- ``A`` is *completed-at* ``Y``.
-
-A memory operation ``B`` *happens-before* an overlapping asynchronous
-operation ``A`` if ``B`` is *program-ordered* before ``A``.
+When a thread executes an async *instruction* ``I``, it initiates a
+corresponding async *operation* ``A``, and ``I`` is said to *happen-before*
+``A``.
-.. note::
+An async operation ``A`` initiated by an instruction ``I`` *happens-before* a
+``wait.asyncmark()`` operation ``Y`` if there exists an ``asyncmark()``
+operation ``X`` such that:
- The *only if* in the above wording implies that unlike the default LLVM
- memory model, certain program order edges are not automatically included in
- ``happens-before``.
+- ``I`` is *program-ordered* before ``X``, and
+- ``X`` is *completed-at* ``Y``.
Examples
========
@@ -238,7 +225,9 @@ being *completed-at* ``D``.
void bar() {
asyncmark(); // B
asyncmark(); // C
+ ...
asyncmark(); // X
+ ... // no wait.asyncmark()
wait.asyncmark(1); // D
}
@@ -273,7 +262,9 @@ examined at ``Y``.
void bar() {
asyncmark(); // B
asyncmark(); // C
+ ... // no asyncmark()
wait.asyncmark(0); // Y
+ ...
wait.asyncmark(1); // D
}
More information about the llvm-commits
mailing list