[llvm] [AMDGPU] Add amdgcn.av.(load|store).b128 intrinsics (PR #191390)

[Sameer Sahasrabuddhe via llvm-commits]

================
@@ -1824,6 +1824,88 @@ The AMDGPU backend implements the following LLVM IR intrinsics.
 
    List AMDGPU intrinsics.
 
+'``llvm.amdgcn.av``' Intrinsics
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The '``llvm.amdgcn.av``' intrinsics perform load and store operations on flat or
+global memory, with explicit control on how their side-effects propagate through
+the system. They take a *scope* argument as a string metadata, which indicates
+the scope within which these side-effects are guaranteed to be observable.
+[TODO: The exact semantics as a memory consistency model is a work in progress.]
+
+The pointer argument can be a global pointer (``addrspace(1)``) or a flat
+pointer (``addrspace(0)``). Global pointers select ``global_load``/
+``global_store`` instructions; flat pointers select ``flat_load``/
+``flat_store`` instructions. The cache policy bits are the same in both cases.
+
+.. code-block:: llvm
+
+   <4 x i32> @llvm.amdgcn.av.load.b128.p1(
+       ptr addrspace(1), ; source (global)
+       metadata)         ; scope    - e.g. '!0' where '!0 = !{!"workgroup"}'
+
+   <4 x i32> @llvm.amdgcn.av.load.b128.p0(
+       ptr,              ; source (flat)
+       metadata)         ; scope
+
+   void @llvm.amdgcn.av.store.b128.p1(
+       ptr addrspace(1), ; destination (global)
+       <4 x i32>,        ; value
+       metadata)         ; scope
+
+   void @llvm.amdgcn.av.store.b128.p0(
+       ptr,              ; destination (flat)
+       <4 x i32>,        ; value
+       metadata)         ; scope
+
+Implementation Details
+++++++++++++++++++++++
+
+This section is informational and for **internal reference only**. Users should
+not rely on the expansions described below. The only reliable user-level
+guarantees are those provided by the memory consistency model, which is
+currently a work in progress.
+
+The tables below show the cache policy bits for global pointer variants.
+Flat pointer variants use the corresponding ``flat_load``/``flat_store``
+instructions with the same cache policy bits.
+
+.. table:: AMDGPU Load-Visible Implementation
+
+   ============== ========================== ========================== ========================== ========================== ==========================
+   targets        instruction                           ``"wavefront"``            ``"workgroup"``                ``"agent"``      ``""`` (empty string)
+   ============== ========================== ========================== ========================== ========================== ==========================
+   gfx90*         ``global_load_dwordx4``                                                                             ``glc``                    ``glc``
+
+   gfx942, gfx950 ``global_load_dwordx4``                        (wave)            ``sc0`` (group)           ``sc1`` (device)       ``sc0 sc1`` (system)
+
+   gfx10*         ``global_load_dwordx4``                                                  ``glc``                ``glc dlc``                ``glc dlc``
+
+   gfx11*         ``global_load_dwordx4``                                                  ``glc``                    ``glc``                    ``glc``
+
+   gfx120*        ``global_load_b128``                             (CU)    ``scope:SCOPE_SE`` (SE)  ``scope:SCOPE_DEV`` (DEV)  ``scope:SCOPE_SYS`` (SYS)
+
+   gfx125*        ``global_load_b128``                             (CU)    ``scope:SCOPE_SE`` (SE)  ``scope:SCOPE_DEV`` (DEV)  ``scope:SCOPE_SYS`` (SYS)
+   ============== ========================== ========================== ========================== ========================== ==========================
----------------
ssahasra wrote:

There's a mismatch in expectations here. 

1. These intrinsics allow the user to ask for LLVM scopes like "workgroup" and "cluster". The do not have a way to express hardware details like "SE" and "CU".
2. Currently, "workgroup" scope is lowered to SE scope because of WGP mode. This is the same as atomics, and also the same as the other intrinsics that we introduced in the ROCm release called `global_store_b128` etc. These are the ones that RCCL currently uses are far as I know, and the new upstream intrinsics will match that performance.
3. It is a performance improvement if the compiler can translate "workgroup" scope more precisely to SE or CU based on WGP mode. But that's a new task to be taken up separately. This PR merely upstreams the same thing that exists in the ROCm release, but with newer names and well-defined semantics.

https://github.com/llvm/llvm-project/pull/191390