[llvm] [AMDGPU] Make `ds_atomic_barrier` operations atomic (PR #194351)

[via llvm-commits]

llvmbot wrote:


<!--LLVM PR SUMMARY COMMENT-->

@llvm/pr-subscribers-backend-amdgpu

Author: Pierre van Houtryve (Pierre-vh)

<details>
<summary>Changes</summary>

Add the MMO, and document them as such in AMDGPUUsage.

---
Full diff: https://github.com/llvm/llvm-project/pull/194351.diff


3 Files Affected:

- (modified) llvm/docs/AMDGPUUsage.rst (+18) 
- (modified) llvm/lib/Target/AMDGPU/SIISelLowering.cpp (+1) 
- (added) llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll (+29) 


``````````diff
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 1f7f1f92f5e2d..323a6cd827aa7 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -1777,6 +1777,24 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
                                                    * :ref:`Synchronization Scope<amdgpu-intrinsics-syncscope-metadata-operand>`.
                                                      Note that the scope used must ensure that the L2 cache will be hit.
 
+  llvm.amdgcn.ds.atomic.barrier.arrive.rtn.b64     Available starting GFX12.5.
+                                                   Corresponds to ``ds_atomic_barrier_arrive_rtn_b64``.
+
+                                                   For the purposes of the memory model, this is a relaxed atomic
+                                                   read-modify-write operation in the local address space.
+                                                   It remains in order with other operations that use ``DScnt``.
+
+  llvm.amdgcn.ds.atomic.async.barrier.arrive.b64   Available starting GFX12.5.
+                                                   Corresponds to ``ds_atomic_async_barrier_arrive_rtn_b64``.
+
+                                                   For the purposes of the memory model, this is an asynchronous
+                                                   relaxed atomic read-modify-write operation in the local address space,
+                                                   and it remains in order with other operations that use ``ASYNCcnt``.
+
+                                                   **NOTE:** Asynchronous operations are currently not
+                                                   covered by the memory model (and thus this operation cannot currently
+                                                   pair with fences); this is a work in progress.
+
   ==============================================   ==========================================================
 
 .. TODO::
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index f08e12a7fbf31..92d9e305f2c3b 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -1552,6 +1552,7 @@ void SITargetLowering::getTgtMemIntrinsic(SmallVectorImpl<IntrinsicInfo> &Infos,
     Info.size = 8;
     Info.align.reset();
     Info.flags = Flags | MachineMemOperand::MOLoad | MachineMemOperand::MOStore;
+    Info.order = AtomicOrdering::Monotonic;
     Infos.push_back(Info);
     return;
   }
diff --git a/llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll b/llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll
new file mode 100644
index 0000000000000..1ba1172d22486
--- /dev/null
+++ b/llvm/test/CodeGen/AMDGPU/lds-barrier-memoperand.ll
@@ -0,0 +1,29 @@
+; NOTE: Assertions have been autogenerated by utils/update_mir_test_checks.py UTC_ARGS: --version 6
+; RUN: llc -global-isel=0 -mtriple=amdgcn -mcpu=gfx1250 -stop-before=si-memory-legalizer < %s | FileCheck --check-prefix=GCN %s
+; RUN: llc -global-isel=1 -mtriple=amdgcn -mcpu=gfx1250 -stop-before=si-memory-legalizer < %s | FileCheck --check-prefix=GCN %s
+
+; Check LDS barrier arrive operations are marked as atomic.
+
+define void @test_ds_atomic_barrier_arrive_rtn_b64(i64 %data, ptr addrspace(3) %bar) {
+  ; GCN-LABEL: name: test_ds_atomic_barrier_arrive_rtn_b64
+  ; GCN: bb.0.entry:
+  ; GCN-NEXT:   liveins: $vgpr0, $vgpr1, $vgpr2
+  ; GCN-NEXT: {{  $}}
+  ; GCN-NEXT:   dead renamable $vgpr0_vgpr1 = DS_ATOMIC_BARRIER_ARRIVE_RTN_B64 killed renamable $vgpr2, killed renamable $vgpr0_vgpr1, 0, 0, implicit $exec :: (load store monotonic (s64) on %ir.bar, addrspace 3)
+  ; GCN-NEXT:   S_SETPC_B64_return undef $sgpr30_sgpr31
+entry:
+  %ret = call i64 @llvm.amdgcn.ds.atomic.barrier.arrive.rtn.b64(ptr addrspace(3) %bar, i64 %data)
+  ret void
+}
+
+define void @test_ds_atomic_async_barrier_arrive_b64(ptr addrspace(3) %bar) {
+  ; GCN-LABEL: name: test_ds_atomic_async_barrier_arrive_b64
+  ; GCN: bb.0.entry:
+  ; GCN-NEXT:   liveins: $vgpr0
+  ; GCN-NEXT: {{  $}}
+  ; GCN-NEXT:   DS_ATOMIC_ASYNC_BARRIER_ARRIVE_B64 killed renamable $vgpr0, 0, 0, implicit-def dead $asynccnt, implicit $exec, implicit $asynccnt :: (load store monotonic (s64) on %ir.bar, addrspace 3)
+  ; GCN-NEXT:   S_SETPC_B64_return undef $sgpr30_sgpr31
+entry:
+  call void @llvm.amdgcn.ds.atomic.async.barrier.arrive.b64(ptr addrspace(3) %bar)
+  ret void
+}

``````````

</details>


https://github.com/llvm/llvm-project/pull/194351