[llvm] [AMDGPU] Make `ds_atomic_barrier` operations atomic (PR #194351)
Pierre van Houtryve via llvm-commits
llvm-commits at lists.llvm.org
Tue Apr 28 01:20:54 PDT 2026
https://github.com/Pierre-vh updated https://github.com/llvm/llvm-project/pull/194351
>From 96c8f3f166d23d762891773498c1ddaa4b4e0b7e Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Mon, 27 Apr 2026 13:43:53 +0200
Subject: [PATCH 1/3] [AMDGPU] Make `ds_atomic_barrier` operations atomic
Add the MMO, and document them as such in AMDGPUUsage.
---
llvm/docs/AMDGPUUsage.rst | 18 ++++++++++++
llvm/lib/Target/AMDGPU/SIISelLowering.cpp | 1 +
.../CodeGen/AMDGPU/lds-barrier-memoperand.ll | 29 +++++++++++++++++++
3 files changed, 48 insertions(+)
create mode 100644 llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 1f7f1f92f5e2d..323a6cd827aa7 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1777,6 +1777,24 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
* :ref:`Synchronization Scope<amdgpu-intrinsics-syncscope-metadata-operand>`.
Note that the scope used must ensure that the L2 cache will be hit.
+ llvm.amdgcn.ds.atomic.barrier.arrive.rtn.b64 Available starting GFX12.5.
+ Corresponds to ``ds_atomic_barrier_arrive_rtn_b64``.
+
+ For the purposes of the memory model, this is a relaxed atomic
+ read-modify-write operation in the local address space.
+ It remains in order with other operations that use ``DScnt``.
+
+ llvm.amdgcn.ds.atomic.async.barrier.arrive.b64 Available starting GFX12.5.
+ Corresponds to ``ds_atomic_async_barrier_arrive_rtn_b64``.
+
+ For the purposes of the memory model, this is an asynchronous
+ relaxed atomic read-modify-write operation in the local address space,
+ and it remains in order with other operations that use ``ASYNCcnt``.
+
+ **NOTE:** Asynchronous operations are currently not
+ covered by the memory model (and thus this operation cannot currently
+ pair with fences); this is a work in progress.
+
============================================== ==========================================================
.. TODO::
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index f08e12a7fbf31..92d9e305f2c3b 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -1552,6 +1552,7 @@ void SITargetLowering::getTgtMemIntrinsic(SmallVectorImpl<IntrinsicInfo> &Infos,
Info.size = 8;
Info.align.reset();
Info.flags = Flags | MachineMemOperand::MOLoad | MachineMemOperand::MOStore;
+ Info.order = AtomicOrdering::Monotonic;
Infos.push_back(Info);
return;
}
diff --git a/llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll b/llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll
new file mode 100644
index 0000000000000..1ba1172d22486
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll
@@ -0,0 +1,29 @@
+; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -global-isel=0 -mtriple=amdgcn -mcpu=gfx1250 -stop-before=si-memory-legalizer < %s | FileCheck --check-prefix=GCN %s
+; RUN: llc -global-isel=1 -mtriple=amdgcn -mcpu=gfx1250 -stop-before=si-memory-legalizer < %s | FileCheck --check-prefix=GCN %s
+
+; Check LDS barrier arrive operations are marked as atomic.
+
+define void @test_ds_atomic_barrier_arrive_rtn_b64(i64 %data, ptr addrspace(3) %bar) {
+ ; GCN-LABEL: name: test_ds_atomic_barrier_arrive_rtn_b64
+ ; GCN: bb.0.entry:
+ ; GCN-NEXT: liveins: $vgpr0, $vgpr1, $vgpr2
+ ; GCN-NEXT: {{ $}}
+ ; GCN-NEXT: dead renamable $vgpr0_vgpr1 = DS_ATOMIC_BARRIER_ARRIVE_RTN_B64 killed renamable $vgpr2, killed renamable $vgpr0_vgpr1, 0, 0, implicit $exec :: (load store monotonic (s64) on %ir.bar, addrspace 3)
+ ; GCN-NEXT: S_SETPC_B64_return undef $sgpr30_sgpr31
+entry:
+ %ret = call i64 @llvm.amdgcn.ds.atomic.barrier.arrive.rtn.b64(ptr addrspace(3) %bar, i64 %data)
+ ret void
+}
+
+define void @test_ds_atomic_async_barrier_arrive_b64(ptr addrspace(3) %bar) {
+ ; GCN-LABEL: name: test_ds_atomic_async_barrier_arrive_b64
+ ; GCN: bb.0.entry:
+ ; GCN-NEXT: liveins: $vgpr0
+ ; GCN-NEXT: {{ $}}
+ ; GCN-NEXT: DS_ATOMIC_ASYNC_BARRIER_ARRIVE_B64 killed renamable $vgpr0, 0, 0, implicit-def dead $asynccnt, implicit $exec, implicit $asynccnt :: (load store monotonic (s64) on %ir.bar, addrspace 3)
+ ; GCN-NEXT: S_SETPC_B64_return undef $sgpr30_sgpr31
+entry:
+ call void @llvm.amdgcn.ds.atomic.async.barrier.arrive.b64(ptr addrspace(3) %bar)
+ ret void
+}
>From 972d2c2ed1b0c620094c247a3b71e264c0a50d53 Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Tue, 28 Apr 2026 10:13:53 +0200
Subject: [PATCH 2/3] Comments
---
llvm/docs/AMDGPUUsage.rst | 13 ++++---------
1 file changed, 4 insertions(+), 9 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 323a6cd827aa7..2769360d65bb7 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1780,20 +1780,15 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
llvm.amdgcn.ds.atomic.barrier.arrive.rtn.b64 Available starting GFX12.5.
Corresponds to ``ds_atomic_barrier_arrive_rtn_b64``.
- For the purposes of the memory model, this is a relaxed atomic
+ For the purposes of the memory model, this is a monotonic atomic
read-modify-write operation in the local address space.
- It remains in order with other operations that use ``DScnt``.
llvm.amdgcn.ds.atomic.async.barrier.arrive.b64 Available starting GFX12.5.
- Corresponds to ``ds_atomic_async_barrier_arrive_rtn_b64``.
+ Corresponds to ``ds_atomic_async_barrier_arrive_b64``.
For the purposes of the memory model, this is an asynchronous
- relaxed atomic read-modify-write operation in the local address space,
- and it remains in order with other operations that use ``ASYNCcnt``.
-
- **NOTE:** Asynchronous operations are currently not
- covered by the memory model (and thus this operation cannot currently
- pair with fences); this is a work in progress.
+ monotonic atomic read-modify-write operation in the local
+ address space.
============================================== ==========================================================
>From b6ed6b654a4506a12e0b69d9f4c547ba7919fef3 Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Tue, 28 Apr 2026 10:20:31 +0200
Subject: [PATCH 3/3] Add operands doc
---
llvm/docs/AMDGPUUsage.rst | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 2769360d65bb7..2c4ff86aa0964 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1783,6 +1783,15 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
For the purposes of the memory model, this is a monotonic atomic
read-modify-write operation in the local address space.
+ This intrinsic has 2 operands:
+
+ * Local pointer to the LDS barrier data.
+ * Update value; the pending count of the barrier will be
+ decremented by this value (generally 1).
+
+ Returns the LDS barrier data as it was before this operation
+ was executed.
+
llvm.amdgcn.ds.atomic.async.barrier.arrive.b64 Available starting GFX12.5.
Corresponds to ``ds_atomic_async_barrier_arrive_b64``.
@@ -1790,6 +1799,10 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
monotonic atomic read-modify-write operation in the local
address space.
+ This intrinsic has 1 operand:
+
+ * Local pointer to the LDS barrier data.
+
============================================== ==========================================================
.. TODO::
More information about the llvm-commits
mailing list