[llvm] [AMDGPU] Sink async DMA out of s_cbranch_execz then-blocks (PR #196374)

[Vigneshwar Jayakumar via llvm-commits]

================
@@ -0,0 +1,253 @@
+//===-- SISinkAsyncDMA.cpp - Sink async DMA out of execz then-blocks ------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+/// \file
+/// LLVM lowers a divergent branch around global_load_async_to_lds /
+/// global_store_async_from_lds with an S_CBRANCH_EXECZ. Fully-masked waves
+/// therefore skip the DMA entirely, which makes the ASYNCcnt observed at the
+/// join point depend on whether the wave took the branch. Software-pipelined
+/// kernels must then use conservative async waitcnts.
+///
+/// This pass sinks each async DMA out of such a then-block into the join
+/// block, immediately before the EXEC-mask restore (the S_OR that ends the
+/// control-flow region). EXEC at the sunk slot is the same masked value that
+/// guarded the then-block, so per-lane behavior is unchanged, but every wave
+/// now issues the DMA and ASYNCcnt becomes deterministic.
+///
+/// This runs right after SILowerControlFlow so the EXEC restore anchor exists,
+/// and before waitcnt insertion so the improved counts can be used.
+///
+/// This is determinism-only: it never relaxes or rewrites a wait (an
+/// s_wait_asynccnt 0 stays 0). Correctness never depends on the transform
+/// firing; with the pass disabled the code is still correct, just conservative.
----------------
VigneshwarJ wrote:

using Asyncmark , the siinsertwaitcnts will determine the right insertion of asyncwaitcnt.

Given this input, which is the software-pipelining shape we care about:

```llvm
entry:
  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %src,
                                                      ptr addrspace(3) %lds, i32 0, i32 0)
  call void @llvm.amdgcn.asyncmark()
  br i1 %cmp, label %prefetch, label %join

prefetch:
  call void @llvm.amdgcn.global.load.async.to.lds.b32(ptr addrspace(1) %s,
                                                      ptr addrspace(3) %l, i32 0, i32 0)
  br label %join

join:
  ; "the marked one is done, the newer one may stay in flight"
  call void @llvm.amdgcn.wait.asyncmark(i16 0)
```

When the pass fires, the guarded DMA is issued by every wave, so the mark can
leave it outstanding:

```asm
;entry
  global_load_async_to_lds_b32 v0, v2, s[0:1]
  ; execz branch  prefetch.....
  ; join
  global_load_async_to_lds_b32 v2, v1, s[0:1]
  s_or_b32 exec_lo, exec_lo, s2
  ; wait_asyncmark(0)
  s_wait_asynccnt 0x1
```
without this pass
```asm
  global_load_async_to_lds_b32 v0, v2, s[0:1]
  ; execz branch prefetch .....
  global_load_async_to_lds_b32 v2, v1, s[0:1]
  ; join
  s_or_b32 exec_lo, exec_lo, s2
  ; wait_asyncmark(0)
  s_wait_asynccnt 0x0
```

https://github.com/llvm/llvm-project/pull/196374