[llvm] [AMDGPU] Improve the description of asyncmark semantics (PR #202579)
Sameer Sahasrabuddhe via llvm-commits
llvm-commits at lists.llvm.org
Wed Jun 10 22:08:58 PDT 2026
https://github.com/ssahasra updated https://github.com/llvm/llvm-project/pull/202579
>From f744f486be194d29659314dc150d327535610033 Mon Sep 17 00:00:00 2001
From: Sameer Sahasrabuddhe <sameer.sahasrabuddhe at amd.com>
Date: Tue, 9 Jun 2026 16:33:16 +0530
Subject: [PATCH 1/2] [AMDGPU] Improve the description of asyncmark semantics
- The semantics of asyncmarks is now definded purely in terms of sequences,
without referring to the implementation.
- The examples incorrectly used (post)dominance. Fixed that with wording in
terms of asyncmark sequences.
---
llvm/docs/AMDGPUAsyncOperations.rst | 164 +++++++++++++++++++---------
1 file changed, 113 insertions(+), 51 deletions(-)
diff --git a/llvm/docs/AMDGPUAsyncOperations.rst b/llvm/docs/AMDGPUAsyncOperations.rst
index 0b8ea0ae77174..90aac42088be2 100644
--- a/llvm/docs/AMDGPUAsyncOperations.rst
+++ b/llvm/docs/AMDGPUAsyncOperations.rst
@@ -13,9 +13,7 @@ Introduction
Asynchronous operations are memory transfers (usually between the global memory
and LDS) that are completed independently at an unspecified scope. A thread that
requests one or more asynchronous transfers can use *asyncmarks* to track
-their completion. The thread waits for each asyncmark to be *completed*, which
-indicates that requests initiated in *program-order* before this asyncmark have also
-completed.
+their completion.
Operations
==========
@@ -57,49 +55,53 @@ memory and LDS memory.
void @llvm.amdgcn.tensor.load.to.lds(...)
void @llvm.amdgcn.tensor.store.from.lds(...)
-Asyncmark Operations
----------------------
+Asyncmarks
+----------
An *asyncmark* in the abstract machine tracks all the async operations that
-are *program-ordered* before that asyncmark. An asyncmark M is said to be *completed*
-only when all async operations *program-ordered* before M are reported by the
-implementation as having finished, and it is said to be *outstanding* otherwise.
-
-Thus we have the following sufficient condition:
-
- An async operation X is *completed* at a program point P if there exists an
- asyncmark M such that X is *program-ordered* before M, M is *program-ordered* before
- P, and M is completed. X is said to be *outstanding* at P otherwise.
+are *program-ordered* before that asyncmark.
The abstract machine maintains a sequence of asyncmarks during the
execution of a function body, which excludes any asyncmarks produced by calls to
other functions encountered in the currently executing function.
+This sequence is called the *current sequence* of that function body.
``@llvm.amdgcn.asyncmark()``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-When executed, inserts an asyncmark in the sequence associated with the
-currently executing function body.
+Appends an asyncmark to the current sequence.
``@llvm.amdgcn.wait.asyncmark(i16 %N)``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Waits until there are at most N outstanding asyncmarks in the sequence associated
-with the currently executing function body.
+Ensures that the length of the current sequence is at most ``N`` by removing
+asyncmarks from the start of the sequence if it is more than ``N``.
Memory Consistency Model
========================
-Each asynchronous operation consists of a non-atomic read on the source and a
-non-atomic write on the destination. Async "LDS DMA" intrinsics result in async
-accesses that guarantee visibility relative to other memory operations as
-follows:
+An ``asyncmark()`` operation ``X`` that inserts an asyncmark ``M`` is
+*completed-at* a ``wait.asyncmark()`` operation ``Y`` in the same function body
+if:
+
+- ``X`` is *program-ordered* before ``Y``, and
+- ``M`` is not in the current sequence after ``Y`` returns.
+
+An async operation ``A`` is *completed-at* a ``wait.asyncmark()`` operation
+``Y`` if there exists an ``asyncmark()`` operation ``X`` such that:
+
+- ``A`` is *program-ordered* before ``X``, and
+- ``X`` is *completed-at* ``Y``.
- An asynchronous operation `A` program ordered before an overlapping memory
- operation `X` happens-before `X` only if `A` is completed before `X`.
+An asynchronous operation ``A`` *happens-before* an overlapping memory operation
+``B`` only if there exists a ``wait.asyncmark()`` operation ``Y`` such that:
- A memory operation `X` program ordered before an overlapping asynchronous
- operation `A` happens-before `A`.
+- ``A`` is *program-ordered* before ``Y``, and
+- ``Y`` is *program-ordered* before ``B``, and
+- ``A`` is *completed-at* ``Y``.
+
+A memory operation ``B`` *happens-before* an overlapping asynchronous
+operation ``A`` if ``B`` is *program-ordered* before ``A``.
.. note::
@@ -192,8 +194,11 @@ Ordinary function call
// third block
asyncmark();
- wait.asyncmark(1); // wait for the second block
- wait.asyncmark(0); // will wait for third block, including bar()
+ // wait for the second block
+ wait.asyncmark(1);
+
+ // wait for the third block, including bar()
+ wait.asyncmark(0);
}
Implementation notes
@@ -201,17 +206,8 @@ Implementation notes
[This section is informational.]
-Optimization
-------------
-
-The implementation may eliminate asyncmark/wait intrinsics in the following cases:
-
-1. An ``asyncmark`` operation which is not included in the wait count of a later
- wait operation in the current function. In particular, an ``asyncmark`` which
- is not post-dominated by any ``wait.asyncmark``.
-2. A ``wait.asyncmark`` whose wait count is more than the outstanding async
- asyncmarks at that point. In particular, a ``wait.asyncmark`` that is not
- dominated by any ``asyncmark``.
+Function Calls
+--------------
In general, at a function call, if the caller uses sufficient waits to track
its own async operations, the actions performed by the callee cannot affect
@@ -220,30 +216,96 @@ correctness. But inlining such a call may result in redundant waits.
.. code-block:: c++
void foo() {
- asyncmark(); // A
+ ...
+ asyncmark(); // X
+ ... // no wait.asyncmark()
}
void bar() {
- asyncmark(); // B
- asyncmark(); // C
+ asyncmark(); // B
+ asyncmark(); // C
foo();
- wait.asyncmark(1);
+ wait.asyncmark(1); // D
+ }
+
+Before inlining, it is unspecified whether ``X`` is *completed-at* ``D``, while
+``C`` is **not** *completed-at* ``D``. The programmer can only rely on ``B``
+being *completed-at* ``D``.
+
+.. code-block:: c++
+
+ void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ asyncmark(); // X
+ wait.asyncmark(1); // D
}
-Before inlining, the ``wait.asyncmark`` waits for asyncmark B to be completed.
+After inlining, ``C`` is also *completed-at* ``D`` and ``X`` is **not**
+*completed-at* ``D``.
+
+Conversely, a ``wait.asyncmark`` call inside a callee cannot be used to track
+asyncmarks inserted by the caller, since this ``wait.asyncmark`` can only
+observe the current sequence of the callee.
.. code-block:: c++
void foo() {
+ ... // no asyncmark()
+ wait.asyncmark(0); // Y
+ ...
}
void bar() {
- asyncmark(); // B
- asyncmark(); // C
- asyncmark(); // A from call to foo()
- wait.asyncmark(1);
+ asyncmark(); // B
+ asyncmark(); // C
+ foo();
+ wait.asyncmark(1); // D
+ }
+
+In the above example, it is unspecified whether ``B`` and ``C`` in ``bar()`` are
+*completed-at* ``Y``, because they are not included in the sequence that can be
+examined at ``Y``.
+
+.. code-block:: c++
+
+ void bar() {
+ asyncmark(); // B
+ asyncmark(); // C
+ wait.asyncmark(0); // Y
+ wait.asyncmark(1); // D
}
-After inlining, the ``wait.asyncmark`` now waits for asyncmark C to complete, which is
-longer than necessary. Ideally, the optimizer should have eliminated asyncmark A in
-the body of foo() itself.
+After inlining, both ``B`` and ``C`` are *completed-at* ``Y``.
+
+Optimization
+------------
+
+The implementation may eliminate asyncmark/wait intrinsics in the following
+cases. These are just examples and not meant to be an exhaustive list.
+
+1. An ``asyncmark`` operation which remains in the current sequence along every
+ path that reaches the function exit.
+
+ .. code-block:: c++
+
+ void foo() {
+ ...
+ asyncmark(); // X
+ ... // no wait.asyncmark()
+ }
+
+ Here, ``X`` can be eliminated.
+
+2. A ``wait.asyncmark`` which sees an empty sequence of asyncmarks along every
+ path that reaches it.
+
+ .. code-block:: c++
+
+ void foo() {
+ ... // no asyncmark()
+ wait.asyncmark(0); // Y
+ ...
+ }
+
+ Here, ``Y`` can be eliminated.
>From 69d4f00fcaa1970e4d2c65481d752538f7531126 Mon Sep 17 00:00:00 2001
From: Sameer Sahasrabuddhe <sameer.sahasrabuddhe at amd.com>
Date: Thu, 11 Jun 2026 10:37:53 +0530
Subject: [PATCH 2/2] tighten the defintion of *completed-at*
---
llvm/docs/AMDGPUAsyncOperations.rst | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/llvm/docs/AMDGPUAsyncOperations.rst b/llvm/docs/AMDGPUAsyncOperations.rst
index 90aac42088be2..0f66df107970b 100644
--- a/llvm/docs/AMDGPUAsyncOperations.rst
+++ b/llvm/docs/AMDGPUAsyncOperations.rst
@@ -85,7 +85,8 @@ An ``asyncmark()`` operation ``X`` that inserts an asyncmark ``M`` is
if:
- ``X`` is *program-ordered* before ``Y``, and
-- ``M`` is not in the current sequence after ``Y`` returns.
+- ``M`` is not in the current sequence at any operation ``Z`` that immediately
+ follows ``Y`` in *program-order*.
An async operation ``A`` is *completed-at* a ``wait.asyncmark()`` operation
``Y`` if there exists an ``asyncmark()`` operation ``X`` such that:
More information about the llvm-commits
mailing list