[llvm-branch-commits] [llvm] [AMDGPU] Make amdgpu.noclobber respect acquire operations (PR #219991)

Fabian Ritter via llvm-branch-commits llvm-branch-commits at lists.llvm.org
Mon Nov 16 02:23:54 PST 2026


https://github.com/ritter-x2a updated https://github.com/llvm/llvm-project/pull/219991

>From e8b2113dd67d4377b800a4cac47e29f431731c93 Mon Sep 17 00:00:00 2001
From: Fabian Ritter <fabian.ritter at amd.com>
Date: Mon, 31 Aug 2026 10:49:48 -0400
Subject: [PATCH 1/6] [AMDGPU] Make amdgpu.noclobber respect acquire operations

AMDGPUAnnotateUniformValues should annotate uniform global loads as
`!amdgpu.noclobber` if the memory they are reading is never clobbered (since
kernel launch) before the load is executed. That allows the backend to use
scalar `s_load_*` instructions instead of vector loads (which would otherwise
be illegal because the scalar cache is not kept coherent with the vector
cache).

So far, loads were annotated if the thread executing the load has not clobbered
the relevant memory location, but changes from concurrent threads that were
acquired via synchronization were ignored (which is unsound).

This patch defines a semantics for `!amdgpu.noclobber` metadata in the
AMDGPUUsage and changes AMDGPUAnnotateUniformValues to respect that semantics.
Now, loads are no longer annotated if they are atomic or if an acquire fence or
load can be executed before them (since they can read from stores in other
threads in these cases).

The patch also fixes an unnecessarily pessimistic case, where atomic stores that
don't alias with the load were considered clobbering.

The noclobber definition uses `undef` instead of `poison` because data races are
currently still defined as `undef`; we need the noclobber definition to use the
same as the data race definition so that it's legal to infer noclobber if the
only way another thread could clobber the memory location would be a data race.

This implementation is overly pessimistic for one-as syncscopes: one-as acquire
operations that don't affect the global addrspace could be ignored for the
noclobber analysis. Improving this is left for a follow-up PR.

For ROCM-29879.
---
 llvm/docs/AMDGPUMemoryModel.md                |   2 +
 llvm/docs/AMDGPUUsage.md                      |  11 +
 .../AMDGPU/AMDGPUAnnotateUniformValues.cpp    |   7 +
 llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp  |  36 +-
 .../AMDGPU/agpr-copy-no-free-registers.ll     | 402 +++++++++---------
 llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll |  93 ++--
 6 files changed, 292 insertions(+), 259 deletions(-)

diff --git a/llvm/docs/AMDGPUMemoryModel.md b/llvm/docs/AMDGPUMemoryModel.md
index 2b247a2e1985fa..d69f7554c6f253 100644
--- a/llvm/docs/AMDGPUMemoryModel.md
+++ b/llvm/docs/AMDGPUMemoryModel.md
@@ -345,6 +345,8 @@ contains `Y`.
 [Informational note --- The "{ref}`contains<amdgpu-subscopes>`" relation is
 reflexive over scope instances.]
 
+(amdgpu-location-order)=
+
 ### Location Order
 
 A write `W` is *location-ordered* before an access `Y` to the same address
diff --git a/llvm/docs/AMDGPUUsage.md b/llvm/docs/AMDGPUUsage.md
index 18945f71e5191c..b5c2d13b24864a 100644
--- a/llvm/docs/AMDGPUUsage.md
+++ b/llvm/docs/AMDGPUUsage.md
@@ -3303,6 +3303,17 @@ The field indicates how much data is fetched on a cache miss. It is only support
 on GFX13. GFX13 only honors it for load and atomic operations. GFX13 does not
 support a 32-byte fetch size so it treats CFS_32B the same as CFS_64B.
 
+(amdgpu_noclobber)=
+
+### '`amdgpu.noclobber`' Metadata
+
+If a `load` instruction marked as `!amdgpu.noclobber` may see any write
+other than the initialization of that location at kernel launch according to the
+{ref}`concurrent memory model<amdgpu-location-order>`, it returns `undef`.
+
+This metadata can allow the AMDGPU compiler backend to emit scalar instead of
+vector load instructions.
+
 ## LLVM IR Attributes
 
 The AMDGPU backend supports the following LLVM IR attributes.
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
index ca66ee7721791c..855b116598fd54 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
@@ -77,6 +77,13 @@ void AMDGPUAnnotateUniformValues::visitLoadInst(LoadInst &I) {
   // for memory operations that are live in to entry points only.
   if (!isEntryFunc)
     return;
+
+  // If I is atomic, it might see the effects of concurrent clobbering accesses,
+  // so local clobber analysis is not sufficient to ensure that it sees the
+  // initial value.
+  if (I.isAtomic())
+    return;
+
   bool GlobalLoad = I.getPointerAddressSpace() == AMDGPUAS::GLOBAL_ADDRESS;
   if (GlobalLoad && !AMDGPU::isClobberedInFunction(&I, MSSA, AA))
     setNoClobberMetadata(&I);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
index 0125a402d0b187..d8c30b64a70add 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
@@ -19,6 +19,7 @@
 #include "llvm/IR/LLVMContext.h"
 #include "llvm/IR/ReplaceConstant.h"
 #include "llvm/Support/AMDGPUAddrSpace.h"
+#include "llvm/Support/AtomicOrdering.h"
 
 #define DEBUG_TYPE "amdgpu-memory-utils"
 
@@ -377,9 +378,6 @@ void removeFnAttrFromReachable(CallGraph &CG, Function *KernelRoot,
 bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA) {
   Instruction *DefInst = Def->getMemoryInst();
 
-  if (isa<FenceInst>(DefInst))
-    return false;
-
   if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(DefInst)) {
     switch (II->getIntrinsicID()) {
     case Intrinsic::amdgcn_s_barrier:
@@ -403,15 +401,29 @@ bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA) {
     }
   }
 
-  // Ignore atomics not aliasing with the original load, any atomic is a
-  // universal MemoryDef from MSSA's point of view too, just like a fence.
-  const auto checkNoAlias = [AA, Ptr](auto I) -> bool {
-    return I && AA->isNoAlias(I->getPointerOperand(), Ptr);
+  // Ignore non-acquire atomics not aliasing with the original load, any atomic
+  // is a universal MemoryDef from MSSA's point of view too, just like a fence.
+  // Acquire (or stronger) fences/atomics act as clobbers because they can bring
+  // in effects from other threads.
+  const auto mayAlias = [AA, Ptr](auto I) -> bool {
+    return !AA->isNoAlias(I->getPointerOperand(), Ptr);
   };
 
-  if (checkNoAlias(dyn_cast<AtomicCmpXchgInst>(DefInst)) ||
-      checkNoAlias(dyn_cast<AtomicRMWInst>(DefInst)))
-    return false;
+  if (const auto *F = dyn_cast<FenceInst>(DefInst))
+    return isAcquireOrStronger(F->getOrdering());
+
+  if (const auto *I = dyn_cast<AtomicRMWInst>(DefInst))
+    return isAcquireOrStronger(I->getOrdering()) || mayAlias(I);
+
+  if (const auto *I = dyn_cast<AtomicCmpXchgInst>(DefInst))
+    return isAcquireOrStronger(I->getMergedOrdering()) || mayAlias(I);
+
+  if (const auto *I = dyn_cast<LoadInst>(DefInst))
+    return isAcquireOrStronger(I->getOrdering());
+
+  // (Atomic) stores that don't alias do not clobber.
+  if (const auto *I = dyn_cast<StoreInst>(DefInst))
+    return mayAlias(I);
 
   return true;
 }
@@ -433,8 +445,8 @@ bool isClobberedInFunction(const LoadInst *Load, MemorySSA *MSSA,
   // case add all Defs to WorkList and continue going up and checking all
   // the definitions of this memory location until the root. When all the
   // defs are exhausted and came to the entry state we have no clobber.
-  // Along the scan ignore barriers and fences which are considered clobbers
-  // by the MemorySSA, but not really writing anything into the memory.
+  // Along the scan ignore barriers which are considered clobbers by the
+  // MemorySSA, but not really writing anything into the memory.
   while (!WorkList.empty()) {
     MemoryAccess *MA = WorkList.pop_back_val();
     if (!Visited.insert(MA).second)
diff --git a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
index d37e7695ff7869..ec25c413a0d604 100644
--- a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
+++ b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
@@ -510,50 +510,52 @@ define void @v32_asm_def_use(float %v0, float %v1) #4 {
 define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg2, i64 %arg3, <2 x half> %arg4, <2 x half> %arg5) #3 {
 ; GFX908-LABEL: introduced_copy_to_sgpr:
 ; GFX908:       ; %bb.0: ; %bb
-; GFX908-NEXT:    global_load_ushort v0, v[0:1], off glc
-; GFX908-NEXT:    s_load_dwordx4 s[0:3], s[8:9], 0x0
-; GFX908-NEXT:    s_load_dwordx2 s[4:5], s[8:9], 0x10
-; GFX908-NEXT:    s_load_dword s7, s[8:9], 0x18
-; GFX908-NEXT:    s_mov_b32 s6, 0
-; GFX908-NEXT:    s_mov_b32 s9, s6
+; GFX908-NEXT:    global_load_ushort v16, v[0:1], off glc
+; GFX908-NEXT:    s_load_dwordx4 s[4:7], s[8:9], 0x0
+; GFX908-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x10
+; GFX908-NEXT:    s_load_dword s0, s[8:9], 0x18
+; GFX908-NEXT:    s_mov_b32 s12, 0
+; GFX908-NEXT:    s_mov_b32 s9, s12
 ; GFX908-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX908-NEXT:    v_cvt_f32_u32_e32 v1, s3
-; GFX908-NEXT:    s_sub_i32 s8, 0, s3
-; GFX908-NEXT:    v_cvt_f32_f16_e32 v12, s7
-; GFX908-NEXT:    s_mov_b64 s[18:19], 0
-; GFX908-NEXT:    v_rcp_f32_e32 v1, v1
-; GFX908-NEXT:    v_mov_b32_e32 v14, 0
-; GFX908-NEXT:    v_mul_f32_e32 v1, 0x4f7ffffe, v1
-; GFX908-NEXT:    v_cvt_u32_f32_e32 v1, v1
-; GFX908-NEXT:    v_readfirstlane_b32 s10, v1
-; GFX908-NEXT:    s_mul_i32 s8, s8, s10
-; GFX908-NEXT:    s_mul_hi_u32 s8, s10, s8
-; GFX908-NEXT:    s_add_i32 s10, s10, s8
-; GFX908-NEXT:    s_mul_hi_u32 s8, s2, s10
-; GFX908-NEXT:    s_mul_i32 s10, s8, s3
-; GFX908-NEXT:    s_sub_i32 s10, s2, s10
-; GFX908-NEXT:    s_add_i32 s11, s8, 1
-; GFX908-NEXT:    s_sub_i32 s12, s10, s3
-; GFX908-NEXT:    s_waitcnt vmcnt(0)
-; GFX908-NEXT:    v_readfirstlane_b32 s2, v0
-; GFX908-NEXT:    s_and_b32 s24, s2, 0xffff
-; GFX908-NEXT:    s_cmp_ge_u32 s10, s3
-; GFX908-NEXT:    s_cselect_b32 s8, s11, s8
-; GFX908-NEXT:    s_cselect_b32 s10, s12, s10
-; GFX908-NEXT:    s_add_i32 s11, s8, 1
-; GFX908-NEXT:    s_cmp_ge_u32 s10, s3
-; GFX908-NEXT:    s_cselect_b32 s8, s11, s8
-; GFX908-NEXT:    s_lshr_b32 s7, s7, 16
-; GFX908-NEXT:    v_cvt_f32_f16_e32 v13, s7
-; GFX908-NEXT:    s_mul_i32 s14, s1, s24
-; GFX908-NEXT:    s_mul_hi_u32 s15, s0, s24
-; GFX908-NEXT:    s_mul_i32 s2, s0, s24
-; GFX908-NEXT:    s_add_i32 s3, s15, s14
-; GFX908-NEXT:    s_lshl_b64 s[10:11], s[0:1], 5
-; GFX908-NEXT:    s_lshl_b64 s[12:13], s[4:5], 5
-; GFX908-NEXT:    s_lshl_b64 s[14:15], s[8:9], 5
-; GFX908-NEXT:    s_lshl_b64 s[16:17], s[2:3], 5
+; GFX908-NEXT:    v_cvt_f32_u32_e32 v0, s7
+; GFX908-NEXT:    s_sub_i32 s1, 0, s7
+; GFX908-NEXT:    v_cvt_f32_f16_e32 v18, s0
+; GFX908-NEXT:    v_mov_b32_e32 v17, 0
+; GFX908-NEXT:    v_rcp_f32_e32 v2, v0
+; GFX908-NEXT:    v_mov_b32_e32 v0, 0
+; GFX908-NEXT:    v_mov_b32_e32 v1, 0
+; GFX908-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
+; GFX908-NEXT:    v_cvt_u32_f32_e32 v2, v2
+; GFX908-NEXT:    v_readfirstlane_b32 s2, v2
+; GFX908-NEXT:    s_mul_i32 s1, s1, s2
+; GFX908-NEXT:    s_mul_hi_u32 s1, s2, s1
+; GFX908-NEXT:    s_add_i32 s2, s2, s1
+; GFX908-NEXT:    s_mul_hi_u32 s1, s6, s2
+; GFX908-NEXT:    s_mul_i32 s2, s1, s7
+; GFX908-NEXT:    s_sub_i32 s2, s6, s2
+; GFX908-NEXT:    s_add_i32 s3, s1, 1
+; GFX908-NEXT:    s_sub_i32 s6, s2, s7
+; GFX908-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX908-NEXT:    s_cselect_b32 s1, s3, s1
+; GFX908-NEXT:    s_cselect_b32 s2, s6, s2
+; GFX908-NEXT:    s_add_i32 s3, s1, 1
+; GFX908-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX908-NEXT:    s_cselect_b32 s8, s3, s1
+; GFX908-NEXT:    s_lshr_b32 s2, s0, 16
+; GFX908-NEXT:    v_cvt_f32_f16_e32 v19, s2
+; GFX908-NEXT:    s_lshl_b64 s[6:7], s[4:5], 5
+; GFX908-NEXT:    s_lshl_b64 s[14:15], s[10:11], 5
 ; GFX908-NEXT:    s_and_b64 s[0:1], exec, s[0:1]
+; GFX908-NEXT:    s_lshl_b64 s[16:17], s[8:9], 5
+; GFX908-NEXT:    s_waitcnt vmcnt(0)
+; GFX908-NEXT:    v_readfirstlane_b32 s2, v16
+; GFX908-NEXT:    s_and_b32 s2, 0xffff, s2
+; GFX908-NEXT:    s_mul_i32 s3, s5, s2
+; GFX908-NEXT:    s_mul_hi_u32 s5, s4, s2
+; GFX908-NEXT:    s_mul_i32 s2, s4, s2
+; GFX908-NEXT:    s_add_i32 s3, s5, s3
+; GFX908-NEXT:    s_lshl_b64 s[18:19], s[2:3], 5
+; GFX908-NEXT:    s_and_b64 s[2:3], exec, 0
 ; GFX908-NEXT:    s_branch .LBB3_2
 ; GFX908-NEXT:  .LBB3_1: ; in Loop: Header=BB3_2 Depth=1
 ; GFX908-NEXT:    s_branch .LBB3_9
@@ -564,135 +566,139 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX908-NEXT:    s_cbranch_vccz .LBB3_9
 ; GFX908-NEXT:  ; %bb.3: ; %bb14
 ; GFX908-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT:    s_load_dwordx2 s[20:21], s[18:19], 0x0
-; GFX908-NEXT:    s_cmp_lt_i32 s5, 0
-; GFX908-NEXT:    s_cselect_b64 s[2:3], -1, 0
-; GFX908-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX908-NEXT:    s_add_u32 s7, s20, 1
-; GFX908-NEXT:    s_addc_u32 s9, s21, 0
-; GFX908-NEXT:    s_mul_hi_u32 s22, s10, s7
-; GFX908-NEXT:    s_mul_i32 s9, s10, s9
-; GFX908-NEXT:    s_mul_i32 s23, s11, s7
-; GFX908-NEXT:    s_add_i32 s9, s22, s9
-; GFX908-NEXT:    s_mul_i32 s25, s10, s7
-; GFX908-NEXT:    s_mov_b32 s7, s6
-; GFX908-NEXT:    s_add_i32 s9, s9, s23
-; GFX908-NEXT:    v_mov_b32_e32 v0, s6
-; GFX908-NEXT:    v_mov_b32_e32 v1, s7
-; GFX908-NEXT:    s_and_b64 s[2:3], exec, s[2:3]
-; GFX908-NEXT:    s_mov_b64 s[22:23], s[12:13]
-; GFX908-NEXT:    v_mov_b32_e32 v2, s6
-; GFX908-NEXT:    v_mov_b32_e32 v3, s7
-; GFX908-NEXT:    v_mov_b32_e32 v4, s6
-; GFX908-NEXT:    v_mov_b32_e32 v5, s7
-; GFX908-NEXT:    v_mov_b32_e32 v6, s6
-; GFX908-NEXT:    v_mov_b32_e32 v7, s7
+; GFX908-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
+; GFX908-NEXT:    s_cmp_lt_i32 s11, 0
+; GFX908-NEXT:    s_mov_b32 s13, s12
+; GFX908-NEXT:    s_cselect_b64 s[4:5], -1, 0
+; GFX908-NEXT:    v_mov_b32_e32 v4, s12
+; GFX908-NEXT:    v_mov_b32_e32 v5, s13
+; GFX908-NEXT:    v_mov_b32_e32 v6, s12
+; GFX908-NEXT:    v_mov_b32_e32 v7, s13
+; GFX908-NEXT:    v_mov_b32_e32 v8, s12
+; GFX908-NEXT:    v_mov_b32_e32 v9, s13
+; GFX908-NEXT:    s_mov_b64 s[20:21], s[14:15]
+; GFX908-NEXT:    v_mov_b32_e32 v11, v5
+; GFX908-NEXT:    v_mov_b32_e32 v10, v4
+; GFX908-NEXT:    s_waitcnt vmcnt(0)
+; GFX908-NEXT:    v_readfirstlane_b32 s9, v2
+; GFX908-NEXT:    v_readfirstlane_b32 s13, v3
+; GFX908-NEXT:    s_add_u32 s9, s9, 1
+; GFX908-NEXT:    s_addc_u32 s13, s13, 0
+; GFX908-NEXT:    s_mul_hi_u32 s22, s6, s9
+; GFX908-NEXT:    s_mul_i32 s13, s6, s13
+; GFX908-NEXT:    s_mul_i32 s23, s7, s9
+; GFX908-NEXT:    s_add_i32 s13, s22, s13
+; GFX908-NEXT:    s_mul_i32 s9, s6, s9
+; GFX908-NEXT:    s_and_b64 s[4:5], exec, s[4:5]
+; GFX908-NEXT:    s_add_i32 s13, s13, s23
 ; GFX908-NEXT:    s_branch .LBB3_5
 ; GFX908-NEXT:  .LBB3_4: ; %bb58
 ; GFX908-NEXT:    ; in Loop: Header=BB3_5 Depth=2
-; GFX908-NEXT:    s_add_u32 s20, s20, s24
-; GFX908-NEXT:    s_addc_u32 s21, s21, 0
-; GFX908-NEXT:    s_add_u32 s22, s22, s16
-; GFX908-NEXT:    s_addc_u32 s23, s23, s17
-; GFX908-NEXT:    s_cmp_lt_i32 s21, 0
-; GFX908-NEXT:    s_cbranch_scc0 .LBB3_1
+; GFX908-NEXT:    v_add_co_u32_sdwa v2, vcc, v2, v16 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; GFX908-NEXT:    v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
+; GFX908-NEXT:    s_add_u32 s20, s20, s18
+; GFX908-NEXT:    s_addc_u32 s21, s21, s19
+; GFX908-NEXT:    v_cmp_gt_i32_e32 vcc, 0, v3
+; GFX908-NEXT:    s_cbranch_vccz .LBB3_1
 ; GFX908-NEXT:  .LBB3_5: ; %bb16
 ; GFX908-NEXT:    ; Parent Loop BB3_2 Depth=1
 ; GFX908-NEXT:    ; => This Inner Loop Header: Depth=2
-; GFX908-NEXT:    s_add_u32 s26, s22, s25
-; GFX908-NEXT:    s_addc_u32 s27, s23, s9
-; GFX908-NEXT:    global_load_dword v16, v14, s[26:27] offset:16 glc
+; GFX908-NEXT:    s_add_u32 s22, s20, s9
+; GFX908-NEXT:    s_addc_u32 s23, s21, s13
+; GFX908-NEXT:    global_load_dword v21, v17, s[22:23] offset:16 glc
 ; GFX908-NEXT:    s_waitcnt vmcnt(0)
-; GFX908-NEXT:    global_load_dword v15, v14, s[26:27] offset:20 glc
+; GFX908-NEXT:    global_load_dword v20, v17, s[22:23] offset:20 glc
 ; GFX908-NEXT:    s_waitcnt vmcnt(0)
-; GFX908-NEXT:    global_load_dword v8, v14, s[26:27] offset:24 glc
+; GFX908-NEXT:    global_load_dword v12, v17, s[22:23] offset:24 glc
 ; GFX908-NEXT:    s_waitcnt vmcnt(0)
-; GFX908-NEXT:    global_load_dword v8, v14, s[26:27] offset:28 glc
+; GFX908-NEXT:    global_load_dword v12, v17, s[22:23] offset:28 glc
 ; GFX908-NEXT:    s_waitcnt vmcnt(0)
-; GFX908-NEXT:    ds_read_b64 v[8:9], v14
-; GFX908-NEXT:    ds_read_b64 v[10:11], v0
-; GFX908-NEXT:    s_mov_b64 vcc, s[2:3]
-; GFX908-NEXT:    ; kill: killed $sgpr26 killed $sgpr27
+; GFX908-NEXT:    ds_read_b64 v[12:13], v17
+; GFX908-NEXT:    ds_read_b64 v[14:15], v0
+; GFX908-NEXT:    s_mov_b64 vcc, s[4:5]
 ; GFX908-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX908-NEXT:    s_cbranch_vccnz .LBB3_4
 ; GFX908-NEXT:  ; %bb.6: ; %bb51
 ; GFX908-NEXT:    ; in Loop: Header=BB3_5 Depth=2
+; GFX908-NEXT:    s_mov_b64 vcc, s[2:3]
+; GFX908-NEXT:    s_cbranch_vccnz .LBB3_8
 ; GFX908-NEXT:  ; %bb.7: ; in Loop: Header=BB3_5 Depth=2
-; GFX908-NEXT:    v_add_f32_e32 v17, v12, v8
-; GFX908-NEXT:    v_add_f32_e32 v18, v13, v9
-; GFX908-NEXT:    v_add_f32_e32 v19, 0, v8
-; GFX908-NEXT:    v_add_f32_e32 v20, 0, v9
-; GFX908-NEXT:    v_fma_mix_f32 v10, v16, 1.0, v10 op_sel_hi:[1,1,0]
-; GFX908-NEXT:    v_fma_mix_f32 v11, v16, 1.0, v11 op_sel:[1,0,0] op_sel_hi:[1,1,0]
-; GFX908-NEXT:    v_fma_mix_f32 v8, v15, 1.0, v8 op_sel_hi:[1,1,0]
-; GFX908-NEXT:    v_fma_mix_f32 v9, v15, 1.0, v9 op_sel:[1,0,0] op_sel_hi:[1,1,0]
-; GFX908-NEXT:    v_add_f32_e32 v1, v1, v18
-; GFX908-NEXT:    v_add_f32_e32 v0, v0, v17
-; GFX908-NEXT:    v_add_f32_e32 v3, v3, v20
-; GFX908-NEXT:    v_add_f32_e32 v2, v2, v19
-; GFX908-NEXT:    v_add_f32_e32 v5, v5, v11
-; GFX908-NEXT:    v_add_f32_e32 v4, v4, v10
-; GFX908-NEXT:    v_add_f32_e32 v7, v7, v9
-; GFX908-NEXT:    v_add_f32_e32 v6, v6, v8
+; GFX908-NEXT:    v_add_f32_e32 v22, v18, v12
+; GFX908-NEXT:    v_add_f32_e32 v23, v19, v13
+; GFX908-NEXT:    v_add_f32_e32 v24, 0, v12
+; GFX908-NEXT:    v_add_f32_e32 v25, 0, v13
+; GFX908-NEXT:    v_fma_mix_f32 v14, v21, 1.0, v14 op_sel_hi:[1,1,0]
+; GFX908-NEXT:    v_fma_mix_f32 v15, v21, 1.0, v15 op_sel:[1,0,0] op_sel_hi:[1,1,0]
+; GFX908-NEXT:    v_fma_mix_f32 v12, v20, 1.0, v12 op_sel_hi:[1,1,0]
+; GFX908-NEXT:    v_fma_mix_f32 v13, v20, 1.0, v13 op_sel:[1,0,0] op_sel_hi:[1,1,0]
+; GFX908-NEXT:    v_add_f32_e32 v5, v5, v23
+; GFX908-NEXT:    v_add_f32_e32 v4, v4, v22
+; GFX908-NEXT:    v_add_f32_e32 v7, v7, v25
+; GFX908-NEXT:    v_add_f32_e32 v6, v6, v24
+; GFX908-NEXT:    v_add_f32_e32 v9, v9, v15
+; GFX908-NEXT:    v_add_f32_e32 v8, v8, v14
+; GFX908-NEXT:    v_add_f32_e32 v11, v11, v13
+; GFX908-NEXT:    v_add_f32_e32 v10, v10, v12
 ; GFX908-NEXT:    s_branch .LBB3_4
-; GFX908-NEXT:  ; %bb.8:
+; GFX908-NEXT:  .LBB3_8: ; in Loop: Header=BB3_2 Depth=1
 ; GFX908-NEXT:    s_cbranch_execnz .LBB3_10
 ; GFX908-NEXT:  .LBB3_9: ; %bb12
 ; GFX908-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT:    s_add_u32 s4, s4, s8
-; GFX908-NEXT:    s_addc_u32 s5, s5, 0
-; GFX908-NEXT:    s_add_u32 s12, s12, s14
-; GFX908-NEXT:    s_addc_u32 s13, s13, s15
+; GFX908-NEXT:    s_add_u32 s10, s10, s8
+; GFX908-NEXT:    s_addc_u32 s11, s11, 0
+; GFX908-NEXT:    s_add_u32 s14, s14, s16
+; GFX908-NEXT:    s_addc_u32 s15, s15, s17
 ; GFX908-NEXT:    s_branch .LBB3_2
 ; GFX908-NEXT:  .LBB3_10: ; %DummyReturnBlock
 ; GFX908-NEXT:    s_endpgm
 ;
 ; GFX90A-LABEL: introduced_copy_to_sgpr:
 ; GFX90A:       ; %bb.0: ; %bb
-; GFX90A-NEXT:    global_load_ushort v1, v[0:1], off glc
-; GFX90A-NEXT:    s_load_dwordx4 s[0:3], s[8:9], 0x0
-; GFX90A-NEXT:    s_load_dwordx2 s[4:5], s[8:9], 0x10
-; GFX90A-NEXT:    s_load_dword s7, s[8:9], 0x18
-; GFX90A-NEXT:    s_mov_b32 s6, 0
-; GFX90A-NEXT:    s_mov_b32 s9, s6
+; GFX90A-NEXT:    global_load_ushort v18, v[0:1], off glc
+; GFX90A-NEXT:    s_load_dwordx4 s[4:7], s[8:9], 0x0
+; GFX90A-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x10
+; GFX90A-NEXT:    s_load_dword s0, s[8:9], 0x18
+; GFX90A-NEXT:    s_mov_b32 s12, 0
+; GFX90A-NEXT:    s_mov_b32 s9, s12
 ; GFX90A-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX90A-NEXT:    v_cvt_f32_u32_e32 v0, s3
-; GFX90A-NEXT:    s_sub_i32 s8, 0, s3
-; GFX90A-NEXT:    s_mov_b64 s[18:19], 0
-; GFX90A-NEXT:    v_mov_b32_e32 v14, 0
-; GFX90A-NEXT:    v_rcp_f32_e32 v0, v0
-; GFX90A-NEXT:    v_mul_f32_e32 v0, 0x4f7ffffe, v0
-; GFX90A-NEXT:    v_cvt_u32_f32_e32 v2, v0
-; GFX90A-NEXT:    v_cvt_f32_f16_e32 v0, s7
-; GFX90A-NEXT:    v_readfirstlane_b32 s10, v2
-; GFX90A-NEXT:    s_mul_i32 s8, s8, s10
-; GFX90A-NEXT:    s_mul_hi_u32 s8, s10, s8
-; GFX90A-NEXT:    s_add_i32 s10, s10, s8
-; GFX90A-NEXT:    s_mul_hi_u32 s8, s2, s10
-; GFX90A-NEXT:    s_mul_i32 s10, s8, s3
-; GFX90A-NEXT:    s_sub_i32 s10, s2, s10
-; GFX90A-NEXT:    s_add_i32 s11, s8, 1
-; GFX90A-NEXT:    s_sub_i32 s12, s10, s3
-; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    v_readfirstlane_b32 s2, v1
-; GFX90A-NEXT:    s_and_b32 s24, s2, 0xffff
-; GFX90A-NEXT:    s_cmp_ge_u32 s10, s3
-; GFX90A-NEXT:    s_cselect_b32 s8, s11, s8
-; GFX90A-NEXT:    s_cselect_b32 s10, s12, s10
-; GFX90A-NEXT:    s_add_i32 s11, s8, 1
-; GFX90A-NEXT:    s_cmp_ge_u32 s10, s3
-; GFX90A-NEXT:    s_cselect_b32 s8, s11, s8
-; GFX90A-NEXT:    s_lshr_b32 s7, s7, 16
-; GFX90A-NEXT:    v_cvt_f32_f16_e32 v1, s7
-; GFX90A-NEXT:    s_mul_i32 s14, s1, s24
-; GFX90A-NEXT:    s_mul_hi_u32 s15, s0, s24
-; GFX90A-NEXT:    s_mul_i32 s2, s0, s24
-; GFX90A-NEXT:    s_add_i32 s3, s15, s14
-; GFX90A-NEXT:    s_lshl_b64 s[10:11], s[0:1], 5
-; GFX90A-NEXT:    s_lshl_b64 s[12:13], s[4:5], 5
-; GFX90A-NEXT:    s_lshl_b64 s[14:15], s[8:9], 5
-; GFX90A-NEXT:    s_lshl_b64 s[16:17], s[2:3], 5
+; GFX90A-NEXT:    v_cvt_f32_u32_e32 v0, s7
+; GFX90A-NEXT:    s_sub_i32 s1, 0, s7
+; GFX90A-NEXT:    v_mov_b32_e32 v19, 0
+; GFX90A-NEXT:    v_rcp_f32_e32 v2, v0
+; GFX90A-NEXT:    v_pk_mov_b32 v[0:1], 0, 0
+; GFX90A-NEXT:    v_mul_f32_e32 v2, 0x4f7ffffe, v2
+; GFX90A-NEXT:    v_cvt_u32_f32_e32 v3, v2
+; GFX90A-NEXT:    v_cvt_f32_f16_e32 v2, s0
+; GFX90A-NEXT:    v_readfirstlane_b32 s2, v3
+; GFX90A-NEXT:    s_mul_i32 s1, s1, s2
+; GFX90A-NEXT:    s_mul_hi_u32 s1, s2, s1
+; GFX90A-NEXT:    s_add_i32 s2, s2, s1
+; GFX90A-NEXT:    s_mul_hi_u32 s1, s6, s2
+; GFX90A-NEXT:    s_mul_i32 s2, s1, s7
+; GFX90A-NEXT:    s_sub_i32 s2, s6, s2
+; GFX90A-NEXT:    s_add_i32 s3, s1, 1
+; GFX90A-NEXT:    s_sub_i32 s6, s2, s7
+; GFX90A-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX90A-NEXT:    s_cselect_b32 s1, s3, s1
+; GFX90A-NEXT:    s_cselect_b32 s2, s6, s2
+; GFX90A-NEXT:    s_add_i32 s3, s1, 1
+; GFX90A-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX90A-NEXT:    s_cselect_b32 s8, s3, s1
+; GFX90A-NEXT:    s_lshr_b32 s2, s0, 16
+; GFX90A-NEXT:    v_cvt_f32_f16_e32 v3, s2
+; GFX90A-NEXT:    s_lshl_b64 s[6:7], s[4:5], 5
+; GFX90A-NEXT:    s_lshl_b64 s[14:15], s[10:11], 5
 ; GFX90A-NEXT:    s_and_b64 s[0:1], exec, s[0:1]
+; GFX90A-NEXT:    s_lshl_b64 s[16:17], s[8:9], 5
+; GFX90A-NEXT:    s_waitcnt vmcnt(0)
+; GFX90A-NEXT:    v_readfirstlane_b32 s2, v18
+; GFX90A-NEXT:    s_and_b32 s2, 0xffff, s2
+; GFX90A-NEXT:    s_mul_i32 s3, s5, s2
+; GFX90A-NEXT:    s_mul_hi_u32 s5, s4, s2
+; GFX90A-NEXT:    s_mul_i32 s2, s4, s2
+; GFX90A-NEXT:    s_add_i32 s3, s5, s3
+; GFX90A-NEXT:    s_lshl_b64 s[18:19], s[2:3], 5
+; GFX90A-NEXT:    s_and_b64 s[2:3], exec, 0
 ; GFX90A-NEXT:    s_branch .LBB3_2
 ; GFX90A-NEXT:  .LBB3_1: ; in Loop: Header=BB3_2 Depth=1
 ; GFX90A-NEXT:    s_branch .LBB3_9
@@ -703,77 +709,81 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX90A-NEXT:    s_cbranch_vccz .LBB3_9
 ; GFX90A-NEXT:  ; %bb.3: ; %bb14
 ; GFX90A-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX90A-NEXT:    s_load_dwordx2 s[20:21], s[18:19], 0x0
-; GFX90A-NEXT:    s_cmp_lt_i32 s5, 0
-; GFX90A-NEXT:    s_cselect_b64 s[2:3], -1, 0
-; GFX90A-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX90A-NEXT:    s_add_u32 s7, s20, 1
-; GFX90A-NEXT:    s_addc_u32 s9, s21, 0
-; GFX90A-NEXT:    s_mul_hi_u32 s22, s10, s7
-; GFX90A-NEXT:    s_mul_i32 s9, s10, s9
-; GFX90A-NEXT:    s_mul_i32 s23, s11, s7
-; GFX90A-NEXT:    s_add_i32 s9, s22, s9
-; GFX90A-NEXT:    s_mul_i32 s25, s10, s7
-; GFX90A-NEXT:    s_mov_b32 s7, s6
-; GFX90A-NEXT:    s_add_i32 s9, s9, s23
-; GFX90A-NEXT:    v_pk_mov_b32 v[2:3], s[6:7], s[6:7] op_sel:[0,1]
-; GFX90A-NEXT:    s_and_b64 s[2:3], exec, s[2:3]
-; GFX90A-NEXT:    s_mov_b64 s[22:23], s[12:13]
-; GFX90A-NEXT:    v_pk_mov_b32 v[4:5], s[6:7], s[6:7] op_sel:[0,1]
-; GFX90A-NEXT:    v_pk_mov_b32 v[6:7], s[6:7], s[6:7] op_sel:[0,1]
-; GFX90A-NEXT:    v_pk_mov_b32 v[8:9], s[6:7], s[6:7] op_sel:[0,1]
+; GFX90A-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off
+; GFX90A-NEXT:    s_cmp_lt_i32 s11, 0
+; GFX90A-NEXT:    s_mov_b32 s13, s12
+; GFX90A-NEXT:    s_cselect_b64 s[4:5], -1, 0
+; GFX90A-NEXT:    v_pk_mov_b32 v[6:7], s[12:13], s[12:13] op_sel:[0,1]
+; GFX90A-NEXT:    v_pk_mov_b32 v[8:9], s[12:13], s[12:13] op_sel:[0,1]
+; GFX90A-NEXT:    v_pk_mov_b32 v[10:11], s[12:13], s[12:13] op_sel:[0,1]
+; GFX90A-NEXT:    s_mov_b64 s[20:21], s[14:15]
+; GFX90A-NEXT:    v_pk_mov_b32 v[12:13], v[6:7], v[6:7] op_sel:[0,1]
+; GFX90A-NEXT:    s_waitcnt vmcnt(0)
+; GFX90A-NEXT:    v_readfirstlane_b32 s9, v4
+; GFX90A-NEXT:    v_readfirstlane_b32 s13, v5
+; GFX90A-NEXT:    s_add_u32 s9, s9, 1
+; GFX90A-NEXT:    s_addc_u32 s13, s13, 0
+; GFX90A-NEXT:    s_mul_hi_u32 s22, s6, s9
+; GFX90A-NEXT:    s_mul_i32 s13, s6, s13
+; GFX90A-NEXT:    s_mul_i32 s23, s7, s9
+; GFX90A-NEXT:    s_add_i32 s13, s22, s13
+; GFX90A-NEXT:    s_mul_i32 s9, s6, s9
+; GFX90A-NEXT:    s_and_b64 s[4:5], exec, s[4:5]
+; GFX90A-NEXT:    s_add_i32 s13, s13, s23
 ; GFX90A-NEXT:    s_branch .LBB3_5
 ; GFX90A-NEXT:  .LBB3_4: ; %bb58
 ; GFX90A-NEXT:    ; in Loop: Header=BB3_5 Depth=2
-; GFX90A-NEXT:    s_add_u32 s20, s20, s24
-; GFX90A-NEXT:    s_addc_u32 s21, s21, 0
-; GFX90A-NEXT:    s_add_u32 s22, s22, s16
-; GFX90A-NEXT:    s_addc_u32 s23, s23, s17
-; GFX90A-NEXT:    s_cmp_lt_i32 s21, 0
-; GFX90A-NEXT:    s_cbranch_scc0 .LBB3_1
+; GFX90A-NEXT:    v_add_co_u32_sdwa v4, vcc, v4, v18 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; GFX90A-NEXT:    v_addc_co_u32_e32 v5, vcc, 0, v5, vcc
+; GFX90A-NEXT:    s_add_u32 s20, s20, s18
+; GFX90A-NEXT:    s_addc_u32 s21, s21, s19
+; GFX90A-NEXT:    v_cmp_gt_i32_e32 vcc, 0, v5
+; GFX90A-NEXT:    s_cbranch_vccz .LBB3_1
 ; GFX90A-NEXT:  .LBB3_5: ; %bb16
 ; GFX90A-NEXT:    ; Parent Loop BB3_2 Depth=1
 ; GFX90A-NEXT:    ; => This Inner Loop Header: Depth=2
-; GFX90A-NEXT:    s_add_u32 s26, s22, s25
-; GFX90A-NEXT:    s_addc_u32 s27, s23, s9
-; GFX90A-NEXT:    global_load_dword v16, v14, s[26:27] offset:16 glc
+; GFX90A-NEXT:    s_add_u32 s22, s20, s9
+; GFX90A-NEXT:    s_addc_u32 s23, s21, s13
+; GFX90A-NEXT:    global_load_dword v21, v19, s[22:23] offset:16 glc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    global_load_dword v15, v14, s[26:27] offset:20 glc
+; GFX90A-NEXT:    global_load_dword v20, v19, s[22:23] offset:20 glc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    global_load_dword v10, v14, s[26:27] offset:24 glc
+; GFX90A-NEXT:    global_load_dword v14, v19, s[22:23] offset:24 glc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    global_load_dword v10, v14, s[26:27] offset:28 glc
+; GFX90A-NEXT:    global_load_dword v14, v19, s[22:23] offset:28 glc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
-; GFX90A-NEXT:    ds_read_b64 v[10:11], v14
-; GFX90A-NEXT:    ds_read_b64 v[12:13], v0
-; GFX90A-NEXT:    s_mov_b64 vcc, s[2:3]
-; GFX90A-NEXT:    ; kill: killed $sgpr26 killed $sgpr27
+; GFX90A-NEXT:    ds_read_b64 v[14:15], v19
+; GFX90A-NEXT:    ds_read_b64 v[16:17], v0
+; GFX90A-NEXT:    s_mov_b64 vcc, s[4:5]
+; GFX90A-NEXT:    ; kill: killed $sgpr22 killed $sgpr23
 ; GFX90A-NEXT:    s_waitcnt lgkmcnt(0)
 ; GFX90A-NEXT:    s_cbranch_vccnz .LBB3_4
 ; GFX90A-NEXT:  ; %bb.6: ; %bb51
 ; GFX90A-NEXT:    ; in Loop: Header=BB3_5 Depth=2
+; GFX90A-NEXT:    s_mov_b64 vcc, s[2:3]
+; GFX90A-NEXT:    s_cbranch_vccnz .LBB3_8
 ; GFX90A-NEXT:  ; %bb.7: ; in Loop: Header=BB3_5 Depth=2
-; GFX90A-NEXT:    v_cvt_f32_f16_sdwa v17, v16 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1
-; GFX90A-NEXT:    v_cvt_f32_f16_e32 v16, v16
-; GFX90A-NEXT:    v_cvt_f32_f16_sdwa v19, v15 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1
-; GFX90A-NEXT:    v_cvt_f32_f16_e32 v18, v15
-; GFX90A-NEXT:    v_pk_add_f32 v[20:21], v[0:1], v[10:11]
-; GFX90A-NEXT:    v_pk_add_f32 v[22:23], v[10:11], 0 op_sel_hi:[1,0]
-; GFX90A-NEXT:    v_pk_add_f32 v[12:13], v[16:17], v[12:13]
-; GFX90A-NEXT:    v_pk_add_f32 v[10:11], v[18:19], v[10:11]
-; GFX90A-NEXT:    v_pk_add_f32 v[2:3], v[2:3], v[20:21]
-; GFX90A-NEXT:    v_pk_add_f32 v[4:5], v[4:5], v[22:23]
-; GFX90A-NEXT:    v_pk_add_f32 v[6:7], v[6:7], v[12:13]
-; GFX90A-NEXT:    v_pk_add_f32 v[8:9], v[8:9], v[10:11]
+; GFX90A-NEXT:    v_cvt_f32_f16_sdwa v23, v21 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1
+; GFX90A-NEXT:    v_cvt_f32_f16_e32 v22, v21
+; GFX90A-NEXT:    v_cvt_f32_f16_sdwa v21, v20 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:WORD_1
+; GFX90A-NEXT:    v_cvt_f32_f16_e32 v20, v20
+; GFX90A-NEXT:    v_pk_add_f32 v[24:25], v[2:3], v[14:15]
+; GFX90A-NEXT:    v_pk_add_f32 v[26:27], v[14:15], 0 op_sel_hi:[1,0]
+; GFX90A-NEXT:    v_pk_add_f32 v[16:17], v[22:23], v[16:17]
+; GFX90A-NEXT:    v_pk_add_f32 v[14:15], v[20:21], v[14:15]
+; GFX90A-NEXT:    v_pk_add_f32 v[6:7], v[6:7], v[24:25]
+; GFX90A-NEXT:    v_pk_add_f32 v[8:9], v[8:9], v[26:27]
+; GFX90A-NEXT:    v_pk_add_f32 v[10:11], v[10:11], v[16:17]
+; GFX90A-NEXT:    v_pk_add_f32 v[12:13], v[12:13], v[14:15]
 ; GFX90A-NEXT:    s_branch .LBB3_4
-; GFX90A-NEXT:  ; %bb.8:
+; GFX90A-NEXT:  .LBB3_8: ; in Loop: Header=BB3_2 Depth=1
 ; GFX90A-NEXT:    s_cbranch_execnz .LBB3_10
 ; GFX90A-NEXT:  .LBB3_9: ; %bb12
 ; GFX90A-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX90A-NEXT:    s_add_u32 s4, s4, s8
-; GFX90A-NEXT:    s_addc_u32 s5, s5, 0
-; GFX90A-NEXT:    s_add_u32 s12, s12, s14
-; GFX90A-NEXT:    s_addc_u32 s13, s13, s15
+; GFX90A-NEXT:    s_add_u32 s10, s10, s8
+; GFX90A-NEXT:    s_addc_u32 s11, s11, 0
+; GFX90A-NEXT:    s_add_u32 s14, s14, s16
+; GFX90A-NEXT:    s_addc_u32 s15, s15, s17
 ; GFX90A-NEXT:    s_branch .LBB3_2
 ; GFX90A-NEXT:  .LBB3_10: ; %DummyReturnBlock
 ; GFX90A-NEXT:    s_endpgm
diff --git a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
index ed5e8bd834eb00..31d7449100457f 100644
--- a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
+++ b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
@@ -17,7 +17,7 @@ define amdgpu_kernel void @simple_barrier(ptr addrspace(1) %arg) {
 ; CHECK-NEXT:    fence syncscope("workgroup") acquire
 ; CHECK-NEXT:    tail call void @llvm.amdgcn.wave.barrier()
 ; CHECK-NEXT:    [[I1:%.*]] = getelementptr inbounds i32, ptr addrspace(1) [[ARG]], i64 1, !amdgpu.uniform [[META0]]
-; CHECK-NEXT:    [[I2:%.*]] = load i32, ptr addrspace(1) [[I1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[I2:%.*]] = load i32, ptr addrspace(1) [[I1]], align 4
 ; CHECK-NEXT:    [[I3:%.*]] = add i32 [[I2]], [[I]]
 ; CHECK-NEXT:    [[I4:%.*]] = getelementptr inbounds i32, ptr addrspace(1) [[ARG]], i64 2
 ; CHECK-NEXT:    store i32 [[I3]], ptr addrspace(1) [[I4]], align 4
@@ -32,10 +32,9 @@ define amdgpu_kernel void @simple_barrier(ptr addrspace(1) %arg) {
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    s_barrier
 ; GCN-NEXT:    ; wave barrier
-; GCN-NEXT:    s_load_dword s3, s[0:1], 0x4
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_add_i32 s2, s3, s2
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[0:1] offset:4
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    v_add_u32_e32 v1, s2, v1
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1] offset:8
 ; GCN-NEXT:    s_endpgm
 bb:
@@ -428,7 +427,7 @@ define amdgpu_kernel void @clobber_by_atomic_load(ptr addrspace(1) %arg) {
 ; CHECK-NEXT:  [[BB:.*:]]
 ; CHECK-NEXT:    [[I:%.*]] = load i32, ptr addrspace(1) [[ARG]], align 4, !amdgpu.noclobber [[META0]]
 ; CHECK-NEXT:    [[GEP:%.*]] = getelementptr inbounds i32, ptr addrspace(1) [[ARG]], i64 2, !amdgpu.uniform [[META0]]
-; CHECK-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[GEP]] seq_cst, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[VAL:%.*]] = load atomic i32, ptr addrspace(1) [[GEP]] seq_cst, align 4
 ; CHECK-NEXT:    [[I1:%.*]] = getelementptr inbounds i32, ptr addrspace(1) [[ARG]], i64 3, !amdgpu.uniform [[META0]]
 ; CHECK-NEXT:    [[I2:%.*]] = load i32, ptr addrspace(1) [[I1]], align 4
 ; CHECK-NEXT:    [[I3:%.*]] = add i32 [[I2]], [[I]]
@@ -606,7 +605,7 @@ define protected amdgpu_kernel void @no_alias_atomic_cmpxchg(ptr addrspace(1) %i
 ; CHECK-SAME: ptr addrspace(1) [[IN:%.*]], ptr addrspace(1) [[OUT:%.*]], i32 [[SWAP:%.*]]) {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[UNUSED:%.*]] = cmpxchg ptr addrspace(3) @LDS, i32 7, i32 [[SWAP]] seq_cst monotonic, align 4
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[IN]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[IN]], align 4
 ; CHECK-NEXT:    store i32 [[LD]], ptr addrspace(1) [[OUT]], align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -620,9 +619,8 @@ define protected amdgpu_kernel void @no_alias_atomic_cmpxchg(ptr addrspace(1) %i
 ; GCN-NEXT:    v_mov_b32_e32 v2, s6
 ; GCN-NEXT:    ds_cmpst_b32 v1, v0, v2
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_load_dword s0, s[0:1], 0x0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v0, s0
+; GCN-NEXT:    global_load_dword v0, v1, s[0:1]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v1, v0, s[2:3]
 ; GCN-NEXT:    s_endpgm
 entry:
@@ -637,7 +635,7 @@ define protected amdgpu_kernel void @no_alias_atomic_rmw(ptr addrspace(1) %in, p
 ; CHECK-SAME: ptr addrspace(1) [[IN:%.*]], ptr addrspace(1) [[OUT:%.*]]) {
 ; CHECK-NEXT:  [[ENTRY:.*:]]
 ; CHECK-NEXT:    [[UNUSED:%.*]] = atomicrmw add ptr addrspace(3) @LDS, i32 5 seq_cst, align 4
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[IN]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[IN]], align 4
 ; CHECK-NEXT:    store i32 [[LD]], ptr addrspace(1) [[OUT]], align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -649,9 +647,8 @@ define protected amdgpu_kernel void @no_alias_atomic_rmw(ptr addrspace(1) %in, p
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    ds_add_u32 v1, v0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_load_dword s0, s[0:1], 0x0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v0, s0
+; GCN-NEXT:    global_load_dword v0, v1, s[0:1]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v1, v0, s[2:3]
 ; GCN-NEXT:    s_endpgm
 entry:
@@ -830,7 +827,7 @@ loop:
 define amdgpu_kernel void @monotonic_load(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1) {
 ; CHECK-LABEL: define amdgpu_kernel void @monotonic_load(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]]) {
-; CHECK-NEXT:    [[LD:%.*]] = load atomic i32, ptr addrspace(1) [[P1]] monotonic, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load atomic i32, ptr addrspace(1) [[P1]] monotonic, align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -902,9 +899,9 @@ define amdgpu_kernel void @fence_release(ptr addrspace(1) noalias %out, ptr addr
 define amdgpu_kernel void @fence_acquire(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1, ptr addrspace(1) noalias %p1na) {
 ; CHECK-LABEL: define amdgpu_kernel void @fence_acquire(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
-; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4
 ; CHECK-NEXT:    fence acquire
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -917,9 +914,8 @@ define amdgpu_kernel void @fence_acquire(ptr addrspace(1) noalias %out, ptr addr
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
 ; GCN-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    buffer_wbinvl1_vol
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(1) %p1na monotonic, align 4
@@ -932,9 +928,9 @@ define amdgpu_kernel void @fence_acquire(ptr addrspace(1) noalias %out, ptr addr
 define amdgpu_kernel void @fence_acq_rel(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1, ptr addrspace(1) noalias %p1na) {
 ; CHECK-LABEL: define amdgpu_kernel void @fence_acq_rel(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
-; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4
 ; CHECK-NEXT:    fence acq_rel
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -947,9 +943,8 @@ define amdgpu_kernel void @fence_acq_rel(ptr addrspace(1) noalias %out, ptr addr
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
 ; GCN-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    buffer_wbinvl1_vol
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(1) %p1na monotonic, align 4
@@ -962,9 +957,9 @@ define amdgpu_kernel void @fence_acq_rel(ptr addrspace(1) noalias %out, ptr addr
 define amdgpu_kernel void @fence_seq_cst(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1, ptr addrspace(1) noalias %p1na) {
 ; CHECK-LABEL: define amdgpu_kernel void @fence_seq_cst(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
-; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4
 ; CHECK-NEXT:    fence seq_cst
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -977,9 +972,8 @@ define amdgpu_kernel void @fence_seq_cst(ptr addrspace(1) noalias %out, ptr addr
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
 ; GCN-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    buffer_wbinvl1_vol
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(1) %p1na monotonic, align 4
@@ -993,7 +987,7 @@ define amdgpu_kernel void @no_alias_store_release(ptr addrspace(1) noalias %out,
 ; CHECK-LABEL: define amdgpu_kernel void @no_alias_store_release(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
 ; CHECK-NEXT:    store atomic i32 1, ptr addrspace(1) [[P1NA]] release, align 4
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1018,7 +1012,7 @@ define amdgpu_kernel void @no_alias_store_release(ptr addrspace(1) noalias %out,
 define amdgpu_kernel void @load_acquire(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1, ptr addrspace(1) noalias %p1na) {
 ; CHECK-LABEL: define amdgpu_kernel void @load_acquire(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
-; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] acquire, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] acquire, align 4
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
@@ -1048,7 +1042,7 @@ define amdgpu_kernel void @no_alias_rmw_acquire(ptr addrspace(1) noalias %out, p
 ; CHECK-LABEL: define amdgpu_kernel void @no_alias_rmw_acquire(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = atomicrmw add ptr addrspace(1) [[P1NA]], i32 1 acquire, align 4
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1076,7 +1070,7 @@ define amdgpu_kernel void @no_alias_rmw_acq_rel(ptr addrspace(1) noalias %out, p
 ; CHECK-LABEL: define amdgpu_kernel void @no_alias_rmw_acq_rel(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = atomicrmw add ptr addrspace(1) [[P1NA]], i32 1 acq_rel, align 4
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1130,7 +1124,7 @@ define amdgpu_kernel void @no_alias_cmpxchg_acquire(ptr addrspace(1) noalias %ou
 ; CHECK-LABEL: define amdgpu_kernel void @no_alias_cmpxchg_acquire(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = cmpxchg ptr addrspace(1) [[P1NA]], i32 1, i32 2 acquire monotonic, align 4
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1159,7 +1153,7 @@ define amdgpu_kernel void @no_alias_cmpxchg_acq_rel(ptr addrspace(1) noalias %ou
 ; CHECK-LABEL: define amdgpu_kernel void @no_alias_cmpxchg_acq_rel(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = cmpxchg ptr addrspace(1) [[P1NA]], i32 1, i32 2 acq_rel monotonic, align 4
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1214,7 +1208,7 @@ define amdgpu_kernel void @no_alias_cmpxchg_release(ptr addrspace(1) noalias %ou
 define amdgpu_kernel void @load_acquire_one_as_same(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1, ptr addrspace(1) noalias %p1na) {
 ; CHECK-LABEL: define amdgpu_kernel void @load_acquire_one_as_same(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
-; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] syncscope("workgroup-one-as") acquire, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] syncscope("workgroup-one-as") acquire, align 4
 ; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
@@ -1268,9 +1262,9 @@ define amdgpu_kernel void @load_acquire_one_as_different(ptr addrspace(1) noalia
 define amdgpu_kernel void @fence_acquire_one_as_same(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1, ptr addrspace(1) noalias %p1na) {
 ; CHECK-LABEL: define amdgpu_kernel void @fence_acquire_one_as_same(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
-; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4
 ; CHECK-NEXT:    fence syncscope("workgroup-one-as") acquire, !mmra [[META1:![0-9]+]]
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1282,9 +1276,8 @@ define amdgpu_kernel void @fence_acquire_one_as_same(ptr addrspace(1) noalias %o
 ; GCN-NEXT:    global_load_dword v1, v0, s[0:1] glc
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
-; GCN-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(1) %p1na monotonic, align 4
@@ -1390,7 +1383,7 @@ define amdgpu_kernel void @fence_acquire_one_as_different(ptr addrspace(1) noali
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(3) [[P3:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(3) [[P3]] monotonic, align 4
 ; CHECK-NEXT:    fence syncscope("workgroup-one-as") acquire, !mmra [[META2:![0-9]+]]
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1403,9 +1396,8 @@ define amdgpu_kernel void @fence_acquire_one_as_different(ptr addrspace(1) noali
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(3) %p3 monotonic, align 4
@@ -1451,19 +1443,18 @@ define amdgpu_kernel void @local_barrier(ptr addrspace(1) noalias %out, ptr addr
 ; CHECK-NEXT:    fence syncscope("workgroup") release, !mmra [[META2]]
 ; CHECK-NEXT:    tail call void @llvm.amdgcn.s.barrier()
 ; CHECK-NEXT:    fence syncscope("workgroup") acquire, !mmra [[META2]]
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
 ; GCN-LABEL: local_barrier:
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_barrier
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    s_barrier
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   fence syncscope("workgroup") release, !mmra !{!"amdgpu-synchronize-as", !"local"}

>From 45bc3ec98b611c52686930b5c45f06517a2b5365 Mon Sep 17 00:00:00 2001
From: Fabian Ritter <fabian.ritter at amd.com>
Date: Mon, 31 Aug 2026 12:17:29 -0400
Subject: [PATCH 2/6] Rename "mayAlias" -> "MayAlias"

---
 llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
index d8c30b64a70add..8a4887afd3e00a 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
@@ -405,7 +405,7 @@ bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA) {
   // is a universal MemoryDef from MSSA's point of view too, just like a fence.
   // Acquire (or stronger) fences/atomics act as clobbers because they can bring
   // in effects from other threads.
-  const auto mayAlias = [AA, Ptr](auto I) -> bool {
+  const auto MayAlias = [AA, Ptr](auto I) -> bool {
     return !AA->isNoAlias(I->getPointerOperand(), Ptr);
   };
 
@@ -413,17 +413,17 @@ bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA) {
     return isAcquireOrStronger(F->getOrdering());
 
   if (const auto *I = dyn_cast<AtomicRMWInst>(DefInst))
-    return isAcquireOrStronger(I->getOrdering()) || mayAlias(I);
+    return isAcquireOrStronger(I->getOrdering()) || MayAlias(I);
 
   if (const auto *I = dyn_cast<AtomicCmpXchgInst>(DefInst))
-    return isAcquireOrStronger(I->getMergedOrdering()) || mayAlias(I);
+    return isAcquireOrStronger(I->getMergedOrdering()) || MayAlias(I);
 
   if (const auto *I = dyn_cast<LoadInst>(DefInst))
     return isAcquireOrStronger(I->getOrdering());
 
   // (Atomic) stores that don't alias do not clobber.
   if (const auto *I = dyn_cast<StoreInst>(DefInst))
-    return mayAlias(I);
+    return MayAlias(I);
 
   return true;
 }

>From 92b45e852767a1dd94afdfed85bf5d27eea0ebfc Mon Sep 17 00:00:00 2001
From: Fabian Ritter <fabian.ritter at amd.com>
Date: Mon, 7 Sep 2026 10:58:54 -0400
Subject: [PATCH 3/6] Don't infer noclobber for volatile accesses.

---
 .../AMDGPU/AMDGPUAnnotateUniformValues.cpp     |  8 ++++----
 .../AMDGPU/GlobalISel/irtranslator-call.ll     | 18 +++++++++---------
 .../CodeGen/AMDGPU/move-to-valu-addsubu64.ll   | 14 +++++++-------
 ...e-to-valu-pseudo-scalar-trans-f16-fake16.ll | 10 +++++-----
 ...e-to-valu-pseudo-scalar-trans-f16-true16.ll | 10 +++++-----
 .../AMDGPU/move-to-valu-pseudo-scalar-trans.ll | 10 +++++-----
 llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll  |  2 +-
 7 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
index 855b116598fd54..34d6e68440714c 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
@@ -78,10 +78,10 @@ void AMDGPUAnnotateUniformValues::visitLoadInst(LoadInst &I) {
   if (!isEntryFunc)
     return;
 
-  // If I is atomic, it might see the effects of concurrent clobbering accesses,
-  // so local clobber analysis is not sufficient to ensure that it sees the
-  // initial value.
-  if (I.isAtomic())
+  // If I is atomic or volatile, it might see the effects of concurrent
+  // clobbering accesses, so local clobber analysis is not sufficient to ensure
+  // that it sees the initial value.
+  if (!I.isSimple())
     return;
 
   bool GlobalLoad = I.getPointerAddressSpace() == AMDGPUAS::GLOBAL_ADDRESS;
diff --git a/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-call.ll b/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-call.ll
index be5158852f6957..e87017bee4684f 100644
--- a/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-call.ll
+++ b/llvm/test/CodeGen/AMDGPU/GlobalISel/irtranslator-call.ll
@@ -409,7 +409,7 @@ define amdgpu_kernel void @test_call_external_void_func_i1_signext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i1) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i1) from `ptr addrspace(1) poison`, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i1) = G_LOAD [[DEF]](p1) :: (volatile load (i1) from `ptr addrspace(1) poison`, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i1_signext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -469,7 +469,7 @@ define amdgpu_kernel void @test_call_external_void_func_i1_zeroext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i1) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i1) from `ptr addrspace(1) poison`, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i1) = G_LOAD [[DEF]](p1) :: (volatile load (i1) from `ptr addrspace(1) poison`, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i1_zeroext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -588,7 +588,7 @@ define amdgpu_kernel void @test_call_external_void_func_i8_signext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i8) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i8) from `ptr addrspace(1) poison`, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i8) = G_LOAD [[DEF]](p1) :: (volatile load (i8) from `ptr addrspace(1) poison`, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i8_signext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -649,7 +649,7 @@ define amdgpu_kernel void @test_call_external_void_func_i8_zeroext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i8) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i8) from `ptr addrspace(1) poison`, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i8) = G_LOAD [[DEF]](p1) :: (volatile load (i8) from `ptr addrspace(1) poison`, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i8_zeroext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -767,7 +767,7 @@ define amdgpu_kernel void @test_call_external_void_func_i16_signext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i16) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i16) from `ptr addrspace(1) poison`, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i16) = G_LOAD [[DEF]](p1) :: (volatile load (i16) from `ptr addrspace(1) poison`, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i16_signext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -827,7 +827,7 @@ define amdgpu_kernel void @test_call_external_void_func_i16_zeroext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i16) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i16) from `ptr addrspace(1) poison`, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i16) = G_LOAD [[DEF]](p1) :: (volatile load (i16) from `ptr addrspace(1) poison`, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i16_zeroext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -1165,7 +1165,7 @@ define amdgpu_kernel void @test_call_external_void_func_i48(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i48) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i48) from `ptr addrspace(1) poison`, align 8, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i48) = G_LOAD [[DEF]](p1) :: (volatile load (i48) from `ptr addrspace(1) poison`, align 8, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i48
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -1227,7 +1227,7 @@ define amdgpu_kernel void @test_call_external_void_func_i48_signext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i48) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i48) from `ptr addrspace(1) poison`, align 8, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i48) = G_LOAD [[DEF]](p1) :: (volatile load (i48) from `ptr addrspace(1) poison`, align 8, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i48_signext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
@@ -1289,7 +1289,7 @@ define amdgpu_kernel void @test_call_external_void_func_i48_zeroext(i32) #0 {
   ; CHECK-NEXT:   [[COPY9:%[0-9]+]]:_(p4) = COPY $sgpr8_sgpr9
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:_(p1) = G_IMPLICIT_DEF
   ; CHECK-NEXT:   [[INT:%[0-9]+]]:_(p4) = G_INTRINSIC intrinsic(@llvm.amdgcn.kernarg.segment.ptr)
-  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i48) = G_LOAD [[DEF]](p1) :: (volatile "amdgpu-noclobber" load (i48) from `ptr addrspace(1) poison`, align 8, addrspace 1)
+  ; CHECK-NEXT:   [[LOAD:%[0-9]+]]:_(i48) = G_LOAD [[DEF]](p1) :: (volatile load (i48) from `ptr addrspace(1) poison`, align 8, addrspace 1)
   ; CHECK-NEXT:   ADJCALLSTACKUP 0, 0, implicit-def dead $scc
   ; CHECK-NEXT:   [[GV:%[0-9]+]]:_(p0) = G_GLOBAL_VALUE @external_void_func_i48_zeroext
   ; CHECK-NEXT:   [[COPY10:%[0-9]+]]:_(p4) = COPY [[COPY8]]
diff --git a/llvm/test/CodeGen/AMDGPU/move-to-valu-addsubu64.ll b/llvm/test/CodeGen/AMDGPU/move-to-valu-addsubu64.ll
index 8d5d2ecc4036d2..a0e95fe7e6212c 100644
--- a/llvm/test/CodeGen/AMDGPU/move-to-valu-addsubu64.ll
+++ b/llvm/test/CodeGen/AMDGPU/move-to-valu-addsubu64.ll
@@ -9,7 +9,7 @@ define amdgpu_kernel void @add_reg_imm(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s64) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s64) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 28744523
   ; CHECK-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 -1395630315
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[S_MOV_B32_1]], %subreg.sub0, [[S_MOV_B32_]], %subreg.sub1
@@ -31,8 +31,8 @@ define amdgpu_kernel void @add_reg_reg(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s64) from %ir.ptr.load, addrspace 1)
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR1:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s64) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s64) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR1:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s64) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_ADD_U:%[0-9]+]]:vreg_64 = V_ADD_U64_PSEUDO [[GLOBAL_LOAD_DWORDX2_SADDR]], [[GLOBAL_LOAD_DWORDX2_SADDR1]], implicit-def dead $vcc_lo, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vreg_64 = COPY [[V_ADD_U]]
   ; CHECK-NEXT:   GLOBAL_STORE_DWORDX2_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s64) into %ir.ptr.load, addrspace 1)
@@ -52,7 +52,7 @@ define amdgpu_kernel void @sub_reg_imm(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s64) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s64) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 -28744524
   ; CHECK-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 1395630315
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[S_MOV_B32_1]], %subreg.sub0, [[S_MOV_B32_]], %subreg.sub1
@@ -74,7 +74,7 @@ define amdgpu_kernel void @sub_imm_reg(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s64) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s64) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[S_MOV_B32_:%[0-9]+]]:sreg_32 = S_MOV_B32 28744523
   ; CHECK-NEXT:   [[S_MOV_B32_1:%[0-9]+]]:sreg_32 = S_MOV_B32 -1395630315
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:sreg_64 = REG_SEQUENCE [[S_MOV_B32_1]], %subreg.sub0, [[S_MOV_B32_]], %subreg.sub1
@@ -96,8 +96,8 @@ define amdgpu_kernel void @sub_reg_reg(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s64) from %ir.ptr.load, addrspace 1)
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR1:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s64) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s64) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORDX2_SADDR1:%[0-9]+]]:vreg_64 = GLOBAL_LOAD_DWORDX2_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s64) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_SUB_U:%[0-9]+]]:vreg_64 = V_SUB_U64_PSEUDO [[GLOBAL_LOAD_DWORDX2_SADDR]], [[GLOBAL_LOAD_DWORDX2_SADDR1]], implicit-def dead $vcc_lo, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vreg_64 = COPY [[V_SUB_U]]
   ; CHECK-NEXT:   GLOBAL_STORE_DWORDX2_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s64) into %ir.ptr.load, addrspace 1)
diff --git a/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-fake16.ll b/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-fake16.ll
index fb1cbf00b85aa3..39ed8363b3b3d0 100644
--- a/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-fake16.ll
+++ b/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-fake16.ll
@@ -9,7 +9,7 @@ define amdgpu_kernel void @exp_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_EXP_F16_fake16_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_EXP_F16_fake16_e64 0, [[GLOBAL_LOAD_USHORT_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_EXP_F16_fake16_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_SHORT_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s16) into %ir.ptr.load, addrspace 1)
@@ -28,7 +28,7 @@ define amdgpu_kernel void @log_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_LOG_F16_fake16_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_LOG_F16_fake16_e64 0, [[GLOBAL_LOAD_USHORT_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_LOG_F16_fake16_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_SHORT_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s16) into %ir.ptr.load, addrspace 1)
@@ -47,7 +47,7 @@ define amdgpu_kernel void @rcp_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_RCP_F16_fake16_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_RCP_F16_fake16_e64 0, [[GLOBAL_LOAD_USHORT_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_RCP_F16_fake16_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_SHORT_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s16) into %ir.ptr.load, addrspace 1)
@@ -66,7 +66,7 @@ define amdgpu_kernel void @rsq_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_RSQ_F16_fake16_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_RSQ_F16_fake16_e64 0, [[GLOBAL_LOAD_USHORT_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_RSQ_F16_fake16_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_SHORT_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s16) into %ir.ptr.load, addrspace 1)
@@ -85,7 +85,7 @@ define amdgpu_kernel void @sqrt_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_USHORT_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_USHORT_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_SQRT_F16_fake16_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_SQRT_F16_fake16_e64 0, [[GLOBAL_LOAD_USHORT_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_SQRT_F16_fake16_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_SHORT_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s16) into %ir.ptr.load, addrspace 1)
diff --git a/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-true16.ll b/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-true16.ll
index 00f0ba985b7890..ca16f96986cfe7 100644
--- a/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-true16.ll
+++ b/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans-f16-true16.ll
@@ -9,7 +9,7 @@ define amdgpu_kernel void @exp_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:vgpr_16 = IMPLICIT_DEF
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vgpr_32 = REG_SEQUENCE [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_]], %subreg.lo16, [[DEF]], %subreg.hi16
   ; CHECK-NEXT:   [[V_EXP_F16_t16_e64_:%[0-9]+]]:vgpr_16 = nofpexcept V_EXP_F16_t16_e64 0, [[REG_SEQUENCE]].lo16, 0, 0, 0, implicit $mode, implicit $exec
@@ -32,7 +32,7 @@ define amdgpu_kernel void @log_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:vgpr_16 = IMPLICIT_DEF
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vgpr_32 = REG_SEQUENCE [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_]], %subreg.lo16, [[DEF]], %subreg.hi16
   ; CHECK-NEXT:   [[V_LOG_F16_t16_e64_:%[0-9]+]]:vgpr_16 = nofpexcept V_LOG_F16_t16_e64 0, [[REG_SEQUENCE]].lo16, 0, 0, 0, implicit $mode, implicit $exec
@@ -55,7 +55,7 @@ define amdgpu_kernel void @rcp_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:vgpr_16 = IMPLICIT_DEF
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vgpr_32 = REG_SEQUENCE [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_]], %subreg.lo16, [[DEF]], %subreg.hi16
   ; CHECK-NEXT:   [[V_RCP_F16_t16_e64_:%[0-9]+]]:vgpr_16 = nofpexcept V_RCP_F16_t16_e64 0, [[REG_SEQUENCE]].lo16, 0, 0, 0, implicit $mode, implicit $exec
@@ -78,7 +78,7 @@ define amdgpu_kernel void @rsq_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:vgpr_16 = IMPLICIT_DEF
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vgpr_32 = REG_SEQUENCE [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_]], %subreg.lo16, [[DEF]], %subreg.hi16
   ; CHECK-NEXT:   [[V_RSQ_F16_t16_e64_:%[0-9]+]]:vgpr_16 = nofpexcept V_RSQ_F16_t16_e64 0, [[REG_SEQUENCE]].lo16, 0, 0, 0, implicit $mode, implicit $exec
@@ -101,7 +101,7 @@ define amdgpu_kernel void @sqrt_f16(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s16) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_:%[0-9]+]]:vgpr_16 = GLOBAL_LOAD_SHORT_D16_SADDR_t16 [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s16) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[DEF:%[0-9]+]]:vgpr_16 = IMPLICIT_DEF
   ; CHECK-NEXT:   [[REG_SEQUENCE:%[0-9]+]]:vgpr_32 = REG_SEQUENCE [[GLOBAL_LOAD_SHORT_D16_SADDR_t16_]], %subreg.lo16, [[DEF]], %subreg.hi16
   ; CHECK-NEXT:   [[V_SQRT_F16_t16_e64_:%[0-9]+]]:vgpr_16 = nofpexcept V_SQRT_F16_t16_e64 0, [[REG_SEQUENCE]].lo16, 0, 0, 0, implicit $mode, implicit $exec
diff --git a/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans.ll b/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans.ll
index 57fb697f56f5e5..3271f2b23c2dce 100644
--- a/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans.ll
+++ b/llvm/test/CodeGen/AMDGPU/move-to-valu-pseudo-scalar-trans.ll
@@ -9,7 +9,7 @@ define amdgpu_kernel void @exp_f32(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s32) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s32) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_EXP_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_EXP_F32_e64 0, [[GLOBAL_LOAD_DWORD_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_EXP_F32_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_DWORD_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s32) into %ir.ptr.load, addrspace 1)
@@ -28,7 +28,7 @@ define amdgpu_kernel void @log_f32(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s32) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s32) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_LOG_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_LOG_F32_e64 0, [[GLOBAL_LOAD_DWORD_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_LOG_F32_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_DWORD_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s32) into %ir.ptr.load, addrspace 1)
@@ -47,7 +47,7 @@ define amdgpu_kernel void @rcp_f32(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s32) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s32) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_RCP_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_RCP_F32_e64 0, [[GLOBAL_LOAD_DWORD_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_RCP_F32_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_DWORD_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s32) into %ir.ptr.load, addrspace 1)
@@ -66,7 +66,7 @@ define amdgpu_kernel void @rsq_f32(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s32) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s32) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_RSQ_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_RSQ_F32_e64 0, [[GLOBAL_LOAD_DWORD_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_RSQ_F32_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_DWORD_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s32) into %ir.ptr.load, addrspace 1)
@@ -85,7 +85,7 @@ define amdgpu_kernel void @sqrt_f32(ptr addrspace(1) %ptr) {
   ; CHECK-NEXT:   [[COPY:%[0-9]+]]:sgpr_64(p4) = COPY $sgpr4_sgpr5
   ; CHECK-NEXT:   [[S_LOAD_DWORDX2_IMM:%[0-9]+]]:sreg_64_xexec_xnull = S_LOAD_DWORDX2_IMM [[COPY]](p4), 36, 0 :: (dereferenceable invariant load (s64) from %ir.ptr.kernarg.offset, align 4, addrspace 4)
   ; CHECK-NEXT:   [[V_MOV_B32_e32_:%[0-9]+]]:vgpr_32 = V_MOV_B32_e32 0, implicit $exec
-  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile "amdgpu-noclobber" load (s32) from %ir.ptr.load, addrspace 1)
+  ; CHECK-NEXT:   [[GLOBAL_LOAD_DWORD_SADDR:%[0-9]+]]:vgpr_32 = GLOBAL_LOAD_DWORD_SADDR [[S_LOAD_DWORDX2_IMM]], [[V_MOV_B32_e32_]], 0, 0, implicit $exec :: (volatile load (s32) from %ir.ptr.load, addrspace 1)
   ; CHECK-NEXT:   [[V_SQRT_F32_e64_:%[0-9]+]]:vgpr_32 = nofpexcept V_SQRT_F32_e64 0, [[GLOBAL_LOAD_DWORD_SADDR]], 0, 0, implicit $mode, implicit $exec
   ; CHECK-NEXT:   [[COPY1:%[0-9]+]]:vgpr_32 = COPY [[V_SQRT_F32_e64_]]
   ; CHECK-NEXT:   GLOBAL_STORE_DWORD_SADDR [[V_MOV_B32_e32_]], [[COPY1]], [[S_LOAD_DWORDX2_IMM]], 0, 0, implicit $exec :: (store (s32) into %ir.ptr.load, addrspace 1)
diff --git a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
index 31d7449100457f..5f2982320127c5 100644
--- a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
+++ b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
@@ -848,7 +848,7 @@ define amdgpu_kernel void @monotonic_load(ptr addrspace(1) noalias %out, ptr add
 define amdgpu_kernel void @volatile_load(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1) {
 ; CHECK-LABEL: define amdgpu_kernel void @volatile_load(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]]) {
-; CHECK-NEXT:    [[LD:%.*]] = load volatile i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load volatile i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;

>From 86187efc15a778424a60248e0e6191eedb1da468 Mon Sep 17 00:00:00 2001
From: Fabian Ritter <fabian.ritter at amd.com>
Date: Wed, 14 Oct 2026 05:19:22 -0400
Subject: [PATCH 4/6] Update check lines for newly added MMRA tests

---
 llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll | 55 +++++++++----------
 1 file changed, 25 insertions(+), 30 deletions(-)

diff --git a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
index 5f2982320127c5..a1fc7006b83e4e 100644
--- a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
+++ b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
@@ -1290,23 +1290,21 @@ define amdgpu_kernel void @fence_acquire_one_as_same(ptr addrspace(1) noalias %o
 define amdgpu_kernel void @fence_acquire_mmra_same(ptr addrspace(1) noalias %out, ptr addrspace(1) %p1, ptr addrspace(1) noalias %p1na) {
 ; CHECK-LABEL: define amdgpu_kernel void @fence_acquire_mmra_same(
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(1) noalias [[P1NA:%.*]]) {
-; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[P1NA]] monotonic, align 4
 ; CHECK-NEXT:    fence syncscope("workgroup") acquire, !mmra [[META1]]
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
 ; GCN-LABEL: fence_acquire_mmra_same:
 ; GCN:       ; %bb.0:
-; GCN-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x34
-; GCN-NEXT:    v_mov_b32_e32 v0, 0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    global_load_dword v1, v0, s[0:1] glc
+; GCN-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x34
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GCN-NEXT:    v_mov_b32_e32 v0, 0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
-; GCN-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[6:7] glc
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(1) %p1na monotonic, align 4
@@ -1321,7 +1319,7 @@ define amdgpu_kernel void @fence_acquire_one_as_flat(ptr addrspace(1) noalias %o
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr noalias [[P1NA:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr [[P1NA]] monotonic, align 4
 ; CHECK-NEXT:    fence syncscope("workgroup-one-as") acquire, !mmra [[META1]]
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1335,9 +1333,8 @@ define amdgpu_kernel void @fence_acquire_one_as_flat(ptr addrspace(1) noalias %o
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
 ; GCN-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(0) %p1na monotonic, align 4
@@ -1352,24 +1349,23 @@ define amdgpu_kernel void @fence_acquire_mmra_flat(ptr addrspace(1) noalias %out
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr noalias [[P1NA:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr [[P1NA]] monotonic, align 4
 ; CHECK-NEXT:    fence syncscope("workgroup") acquire, !mmra [[META1]]
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
 ; GCN-LABEL: fence_acquire_mmra_flat:
 ; GCN:       ; %bb.0:
-; GCN-NEXT:    s_load_dwordx2 s[0:1], s[4:5], 0x34
+; GCN-NEXT:    s_load_dwordx2 s[6:7], s[4:5], 0x34
+; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
+; GCN-NEXT:    v_mov_b32_e32 v2, 0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v0, s0
-; GCN-NEXT:    v_mov_b32_e32 v1, s1
+; GCN-NEXT:    v_mov_b32_e32 v0, s6
+; GCN-NEXT:    v_mov_b32_e32 v1, s7
 ; GCN-NEXT:    flat_load_dword v0, v[0:1] glc
-; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GCN-NEXT:    s_waitcnt vmcnt(0) lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v0, 0
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
-; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
+; GCN-NEXT:    global_load_dword v0, v2, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
+; GCN-NEXT:    global_store_dword v2, v0, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(0) %p1na monotonic, align 4
   fence syncscope("workgroup") acquire, !mmra !{!"amdgpu-synchronize-as", !"global"}
@@ -1412,7 +1408,7 @@ define amdgpu_kernel void @fence_acquire_mmra_different(ptr addrspace(1) noalias
 ; CHECK-SAME: ptr addrspace(1) noalias [[OUT:%.*]], ptr addrspace(1) [[P1:%.*]], ptr addrspace(3) [[P3:%.*]]) {
 ; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(3) [[P3]] monotonic, align 4
 ; CHECK-NEXT:    fence syncscope("workgroup") acquire, !mmra [[META2]]
-; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4, !amdgpu.noclobber [[META0]]
+; CHECK-NEXT:    [[LD:%.*]] = load i32, ptr addrspace(1) [[P1]], align 4
 ; CHECK-NEXT:    store atomic i32 [[LD]], ptr addrspace(1) [[OUT]] seq_cst, align 4
 ; CHECK-NEXT:    ret void
 ;
@@ -1420,14 +1416,13 @@ define amdgpu_kernel void @fence_acquire_mmra_different(ptr addrspace(1) noalias
 ; GCN:       ; %bb.0:
 ; GCN-NEXT:    s_load_dword s6, s[4:5], 0x34
 ; GCN-NEXT:    s_load_dwordx4 s[0:3], s[4:5], 0x24
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v0, s6
-; GCN-NEXT:    ds_read_b32 v0, v0
-; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    s_load_dword s2, s[2:3], 0x0
 ; GCN-NEXT:    v_mov_b32_e32 v0, 0
 ; GCN-NEXT:    s_waitcnt lgkmcnt(0)
-; GCN-NEXT:    v_mov_b32_e32 v1, s2
+; GCN-NEXT:    v_mov_b32_e32 v1, s6
+; GCN-NEXT:    ds_read_b32 v1, v1
+; GCN-NEXT:    s_waitcnt lgkmcnt(0)
+; GCN-NEXT:    global_load_dword v1, v0, s[2:3]
+; GCN-NEXT:    s_waitcnt vmcnt(0)
 ; GCN-NEXT:    global_store_dword v0, v1, s[0:1]
 ; GCN-NEXT:    s_endpgm
   %acq = load atomic i32, ptr addrspace(3) %p3 monotonic, align 4

>From 2379a7499bb798d2426d74d8b8d58407beaad095 Mon Sep 17 00:00:00 2001
From: Fabian Ritter <fabian.ritter at amd.com>
Date: Thu, 5 Nov 2026 06:10:39 -0500
Subject: [PATCH 5/6] Update comments, regenerate tests

---
 llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp  |   4 +-
 .../AMDGPU/agpr-copy-no-free-registers.ll     | 146 +++++++++---------
 llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll |   6 +-
 .../AMDGPU/promote-kernel-arguments.ll        |   8 +-
 4 files changed, 82 insertions(+), 82 deletions(-)

diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
index 8a4887afd3e00a..c495e2a8d46f22 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
@@ -401,8 +401,8 @@ bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA) {
     }
   }
 
-  // Ignore non-acquire atomics not aliasing with the original load, any atomic
-  // is a universal MemoryDef from MSSA's point of view too, just like a fence.
+  // Ignore non-acquire fences/atomics not aliasing with the original load; MSSA
+  // counts any atomic or fence as a universal MemoryDef.
   // Acquire (or stronger) fences/atomics act as clobbers because they can bring
   // in effects from other threads.
   const auto MayAlias = [AA, Ptr](auto I) -> bool {
diff --git a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
index ec25c413a0d604..81e82618180355 100644
--- a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
+++ b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
@@ -511,14 +511,14 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX908-LABEL: introduced_copy_to_sgpr:
 ; GFX908:       ; %bb.0: ; %bb
 ; GFX908-NEXT:    global_load_ushort v16, v[0:1], off glc
-; GFX908-NEXT:    s_load_dwordx4 s[4:7], s[8:9], 0x0
-; GFX908-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x10
+; GFX908-NEXT:    s_load_dwordx4 s[20:23], s[8:9], 0x0
+; GFX908-NEXT:    s_load_dwordx2 s[6:7], s[8:9], 0x10
 ; GFX908-NEXT:    s_load_dword s0, s[8:9], 0x18
-; GFX908-NEXT:    s_mov_b32 s12, 0
-; GFX908-NEXT:    s_mov_b32 s9, s12
+; GFX908-NEXT:    s_mov_b32 s10, 0
+; GFX908-NEXT:    s_mov_b32 s9, s10
 ; GFX908-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX908-NEXT:    v_cvt_f32_u32_e32 v0, s7
-; GFX908-NEXT:    s_sub_i32 s1, 0, s7
+; GFX908-NEXT:    v_cvt_f32_u32_e32 v0, s23
+; GFX908-NEXT:    s_sub_i32 s1, 0, s23
 ; GFX908-NEXT:    v_cvt_f32_f16_e32 v18, s0
 ; GFX908-NEXT:    v_mov_b32_e32 v17, 0
 ; GFX908-NEXT:    v_rcp_f32_e32 v2, v0
@@ -530,30 +530,30 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX908-NEXT:    s_mul_i32 s1, s1, s2
 ; GFX908-NEXT:    s_mul_hi_u32 s1, s2, s1
 ; GFX908-NEXT:    s_add_i32 s2, s2, s1
-; GFX908-NEXT:    s_mul_hi_u32 s1, s6, s2
-; GFX908-NEXT:    s_mul_i32 s2, s1, s7
-; GFX908-NEXT:    s_sub_i32 s2, s6, s2
+; GFX908-NEXT:    s_mul_hi_u32 s1, s22, s2
+; GFX908-NEXT:    s_mul_i32 s2, s1, s23
+; GFX908-NEXT:    s_sub_i32 s2, s22, s2
 ; GFX908-NEXT:    s_add_i32 s3, s1, 1
-; GFX908-NEXT:    s_sub_i32 s6, s2, s7
-; GFX908-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX908-NEXT:    s_sub_i32 s4, s2, s23
+; GFX908-NEXT:    s_cmp_ge_u32 s2, s23
 ; GFX908-NEXT:    s_cselect_b32 s1, s3, s1
-; GFX908-NEXT:    s_cselect_b32 s2, s6, s2
+; GFX908-NEXT:    s_cselect_b32 s2, s4, s2
 ; GFX908-NEXT:    s_add_i32 s3, s1, 1
-; GFX908-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX908-NEXT:    s_cmp_ge_u32 s2, s23
 ; GFX908-NEXT:    s_cselect_b32 s8, s3, s1
 ; GFX908-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX908-NEXT:    v_cvt_f32_f16_e32 v19, s2
-; GFX908-NEXT:    s_lshl_b64 s[6:7], s[4:5], 5
-; GFX908-NEXT:    s_lshl_b64 s[14:15], s[10:11], 5
+; GFX908-NEXT:    s_lshl_b64 s[12:13], s[20:21], 5
+; GFX908-NEXT:    s_lshl_b64 s[14:15], s[6:7], 5
 ; GFX908-NEXT:    s_and_b64 s[0:1], exec, s[0:1]
 ; GFX908-NEXT:    s_lshl_b64 s[16:17], s[8:9], 5
 ; GFX908-NEXT:    s_waitcnt vmcnt(0)
 ; GFX908-NEXT:    v_readfirstlane_b32 s2, v16
 ; GFX908-NEXT:    s_and_b32 s2, 0xffff, s2
-; GFX908-NEXT:    s_mul_i32 s3, s5, s2
-; GFX908-NEXT:    s_mul_hi_u32 s5, s4, s2
-; GFX908-NEXT:    s_mul_i32 s2, s4, s2
-; GFX908-NEXT:    s_add_i32 s3, s5, s3
+; GFX908-NEXT:    s_mul_i32 s3, s21, s2
+; GFX908-NEXT:    s_mul_hi_u32 s4, s20, s2
+; GFX908-NEXT:    s_mul_i32 s2, s20, s2
+; GFX908-NEXT:    s_add_i32 s3, s4, s3
 ; GFX908-NEXT:    s_lshl_b64 s[18:19], s[2:3], 5
 ; GFX908-NEXT:    s_and_b64 s[2:3], exec, 0
 ; GFX908-NEXT:    s_branch .LBB3_2
@@ -567,30 +567,30 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX908-NEXT:  ; %bb.3: ; %bb14
 ; GFX908-NEXT:    ; in Loop: Header=BB3_2 Depth=1
 ; GFX908-NEXT:    global_load_dwordx2 v[2:3], v[0:1], off
-; GFX908-NEXT:    s_cmp_lt_i32 s11, 0
-; GFX908-NEXT:    s_mov_b32 s13, s12
+; GFX908-NEXT:    s_cmp_lt_i32 s7, 0
+; GFX908-NEXT:    s_mov_b32 s11, s10
 ; GFX908-NEXT:    s_cselect_b64 s[4:5], -1, 0
-; GFX908-NEXT:    v_mov_b32_e32 v4, s12
-; GFX908-NEXT:    v_mov_b32_e32 v5, s13
-; GFX908-NEXT:    v_mov_b32_e32 v6, s12
-; GFX908-NEXT:    v_mov_b32_e32 v7, s13
-; GFX908-NEXT:    v_mov_b32_e32 v8, s12
-; GFX908-NEXT:    v_mov_b32_e32 v9, s13
+; GFX908-NEXT:    v_mov_b32_e32 v4, s10
+; GFX908-NEXT:    v_mov_b32_e32 v5, s11
+; GFX908-NEXT:    v_mov_b32_e32 v6, s10
+; GFX908-NEXT:    v_mov_b32_e32 v7, s11
+; GFX908-NEXT:    v_mov_b32_e32 v8, s10
+; GFX908-NEXT:    v_mov_b32_e32 v9, s11
 ; GFX908-NEXT:    s_mov_b64 s[20:21], s[14:15]
 ; GFX908-NEXT:    v_mov_b32_e32 v11, v5
 ; GFX908-NEXT:    v_mov_b32_e32 v10, v4
 ; GFX908-NEXT:    s_waitcnt vmcnt(0)
 ; GFX908-NEXT:    v_readfirstlane_b32 s9, v2
-; GFX908-NEXT:    v_readfirstlane_b32 s13, v3
+; GFX908-NEXT:    v_readfirstlane_b32 s11, v3
 ; GFX908-NEXT:    s_add_u32 s9, s9, 1
-; GFX908-NEXT:    s_addc_u32 s13, s13, 0
-; GFX908-NEXT:    s_mul_hi_u32 s22, s6, s9
-; GFX908-NEXT:    s_mul_i32 s13, s6, s13
-; GFX908-NEXT:    s_mul_i32 s23, s7, s9
-; GFX908-NEXT:    s_add_i32 s13, s22, s13
-; GFX908-NEXT:    s_mul_i32 s9, s6, s9
+; GFX908-NEXT:    s_addc_u32 s11, s11, 0
+; GFX908-NEXT:    s_mul_hi_u32 s22, s12, s9
+; GFX908-NEXT:    s_mul_i32 s11, s12, s11
+; GFX908-NEXT:    s_mul_i32 s23, s13, s9
+; GFX908-NEXT:    s_add_i32 s11, s22, s11
+; GFX908-NEXT:    s_mul_i32 s9, s12, s9
 ; GFX908-NEXT:    s_and_b64 s[4:5], exec, s[4:5]
-; GFX908-NEXT:    s_add_i32 s13, s13, s23
+; GFX908-NEXT:    s_add_i32 s11, s11, s23
 ; GFX908-NEXT:    s_branch .LBB3_5
 ; GFX908-NEXT:  .LBB3_4: ; %bb58
 ; GFX908-NEXT:    ; in Loop: Header=BB3_5 Depth=2
@@ -604,7 +604,7 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX908-NEXT:    ; Parent Loop BB3_2 Depth=1
 ; GFX908-NEXT:    ; => This Inner Loop Header: Depth=2
 ; GFX908-NEXT:    s_add_u32 s22, s20, s9
-; GFX908-NEXT:    s_addc_u32 s23, s21, s13
+; GFX908-NEXT:    s_addc_u32 s23, s21, s11
 ; GFX908-NEXT:    global_load_dword v21, v17, s[22:23] offset:16 glc
 ; GFX908-NEXT:    s_waitcnt vmcnt(0)
 ; GFX908-NEXT:    global_load_dword v20, v17, s[22:23] offset:20 glc
@@ -644,8 +644,8 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX908-NEXT:    s_cbranch_execnz .LBB3_10
 ; GFX908-NEXT:  .LBB3_9: ; %bb12
 ; GFX908-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT:    s_add_u32 s10, s10, s8
-; GFX908-NEXT:    s_addc_u32 s11, s11, 0
+; GFX908-NEXT:    s_add_u32 s6, s6, s8
+; GFX908-NEXT:    s_addc_u32 s7, s7, 0
 ; GFX908-NEXT:    s_add_u32 s14, s14, s16
 ; GFX908-NEXT:    s_addc_u32 s15, s15, s17
 ; GFX908-NEXT:    s_branch .LBB3_2
@@ -655,14 +655,14 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX90A-LABEL: introduced_copy_to_sgpr:
 ; GFX90A:       ; %bb.0: ; %bb
 ; GFX90A-NEXT:    global_load_ushort v18, v[0:1], off glc
-; GFX90A-NEXT:    s_load_dwordx4 s[4:7], s[8:9], 0x0
-; GFX90A-NEXT:    s_load_dwordx2 s[10:11], s[8:9], 0x10
+; GFX90A-NEXT:    s_load_dwordx4 s[20:23], s[8:9], 0x0
+; GFX90A-NEXT:    s_load_dwordx2 s[6:7], s[8:9], 0x10
 ; GFX90A-NEXT:    s_load_dword s0, s[8:9], 0x18
-; GFX90A-NEXT:    s_mov_b32 s12, 0
-; GFX90A-NEXT:    s_mov_b32 s9, s12
+; GFX90A-NEXT:    s_mov_b32 s10, 0
+; GFX90A-NEXT:    s_mov_b32 s9, s10
 ; GFX90A-NEXT:    s_waitcnt lgkmcnt(0)
-; GFX90A-NEXT:    v_cvt_f32_u32_e32 v0, s7
-; GFX90A-NEXT:    s_sub_i32 s1, 0, s7
+; GFX90A-NEXT:    v_cvt_f32_u32_e32 v0, s23
+; GFX90A-NEXT:    s_sub_i32 s1, 0, s23
 ; GFX90A-NEXT:    v_mov_b32_e32 v19, 0
 ; GFX90A-NEXT:    v_rcp_f32_e32 v2, v0
 ; GFX90A-NEXT:    v_pk_mov_b32 v[0:1], 0, 0
@@ -673,30 +673,30 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX90A-NEXT:    s_mul_i32 s1, s1, s2
 ; GFX90A-NEXT:    s_mul_hi_u32 s1, s2, s1
 ; GFX90A-NEXT:    s_add_i32 s2, s2, s1
-; GFX90A-NEXT:    s_mul_hi_u32 s1, s6, s2
-; GFX90A-NEXT:    s_mul_i32 s2, s1, s7
-; GFX90A-NEXT:    s_sub_i32 s2, s6, s2
+; GFX90A-NEXT:    s_mul_hi_u32 s1, s22, s2
+; GFX90A-NEXT:    s_mul_i32 s2, s1, s23
+; GFX90A-NEXT:    s_sub_i32 s2, s22, s2
 ; GFX90A-NEXT:    s_add_i32 s3, s1, 1
-; GFX90A-NEXT:    s_sub_i32 s6, s2, s7
-; GFX90A-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX90A-NEXT:    s_sub_i32 s4, s2, s23
+; GFX90A-NEXT:    s_cmp_ge_u32 s2, s23
 ; GFX90A-NEXT:    s_cselect_b32 s1, s3, s1
-; GFX90A-NEXT:    s_cselect_b32 s2, s6, s2
+; GFX90A-NEXT:    s_cselect_b32 s2, s4, s2
 ; GFX90A-NEXT:    s_add_i32 s3, s1, 1
-; GFX90A-NEXT:    s_cmp_ge_u32 s2, s7
+; GFX90A-NEXT:    s_cmp_ge_u32 s2, s23
 ; GFX90A-NEXT:    s_cselect_b32 s8, s3, s1
 ; GFX90A-NEXT:    s_lshr_b32 s2, s0, 16
 ; GFX90A-NEXT:    v_cvt_f32_f16_e32 v3, s2
-; GFX90A-NEXT:    s_lshl_b64 s[6:7], s[4:5], 5
-; GFX90A-NEXT:    s_lshl_b64 s[14:15], s[10:11], 5
+; GFX90A-NEXT:    s_lshl_b64 s[12:13], s[20:21], 5
+; GFX90A-NEXT:    s_lshl_b64 s[14:15], s[6:7], 5
 ; GFX90A-NEXT:    s_and_b64 s[0:1], exec, s[0:1]
 ; GFX90A-NEXT:    s_lshl_b64 s[16:17], s[8:9], 5
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
 ; GFX90A-NEXT:    v_readfirstlane_b32 s2, v18
 ; GFX90A-NEXT:    s_and_b32 s2, 0xffff, s2
-; GFX90A-NEXT:    s_mul_i32 s3, s5, s2
-; GFX90A-NEXT:    s_mul_hi_u32 s5, s4, s2
-; GFX90A-NEXT:    s_mul_i32 s2, s4, s2
-; GFX90A-NEXT:    s_add_i32 s3, s5, s3
+; GFX90A-NEXT:    s_mul_i32 s3, s21, s2
+; GFX90A-NEXT:    s_mul_hi_u32 s4, s20, s2
+; GFX90A-NEXT:    s_mul_i32 s2, s20, s2
+; GFX90A-NEXT:    s_add_i32 s3, s4, s3
 ; GFX90A-NEXT:    s_lshl_b64 s[18:19], s[2:3], 5
 ; GFX90A-NEXT:    s_and_b64 s[2:3], exec, 0
 ; GFX90A-NEXT:    s_branch .LBB3_2
@@ -710,26 +710,26 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX90A-NEXT:  ; %bb.3: ; %bb14
 ; GFX90A-NEXT:    ; in Loop: Header=BB3_2 Depth=1
 ; GFX90A-NEXT:    global_load_dwordx2 v[4:5], v[0:1], off
-; GFX90A-NEXT:    s_cmp_lt_i32 s11, 0
-; GFX90A-NEXT:    s_mov_b32 s13, s12
+; GFX90A-NEXT:    s_cmp_lt_i32 s7, 0
+; GFX90A-NEXT:    s_mov_b32 s11, s10
 ; GFX90A-NEXT:    s_cselect_b64 s[4:5], -1, 0
-; GFX90A-NEXT:    v_pk_mov_b32 v[6:7], s[12:13], s[12:13] op_sel:[0,1]
-; GFX90A-NEXT:    v_pk_mov_b32 v[8:9], s[12:13], s[12:13] op_sel:[0,1]
-; GFX90A-NEXT:    v_pk_mov_b32 v[10:11], s[12:13], s[12:13] op_sel:[0,1]
+; GFX90A-NEXT:    v_pk_mov_b32 v[6:7], s[10:11], s[10:11] op_sel:[0,1]
+; GFX90A-NEXT:    v_pk_mov_b32 v[8:9], s[10:11], s[10:11] op_sel:[0,1]
+; GFX90A-NEXT:    v_pk_mov_b32 v[10:11], s[10:11], s[10:11] op_sel:[0,1]
 ; GFX90A-NEXT:    s_mov_b64 s[20:21], s[14:15]
 ; GFX90A-NEXT:    v_pk_mov_b32 v[12:13], v[6:7], v[6:7] op_sel:[0,1]
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
 ; GFX90A-NEXT:    v_readfirstlane_b32 s9, v4
-; GFX90A-NEXT:    v_readfirstlane_b32 s13, v5
+; GFX90A-NEXT:    v_readfirstlane_b32 s11, v5
 ; GFX90A-NEXT:    s_add_u32 s9, s9, 1
-; GFX90A-NEXT:    s_addc_u32 s13, s13, 0
-; GFX90A-NEXT:    s_mul_hi_u32 s22, s6, s9
-; GFX90A-NEXT:    s_mul_i32 s13, s6, s13
-; GFX90A-NEXT:    s_mul_i32 s23, s7, s9
-; GFX90A-NEXT:    s_add_i32 s13, s22, s13
-; GFX90A-NEXT:    s_mul_i32 s9, s6, s9
+; GFX90A-NEXT:    s_addc_u32 s11, s11, 0
+; GFX90A-NEXT:    s_mul_hi_u32 s22, s12, s9
+; GFX90A-NEXT:    s_mul_i32 s11, s12, s11
+; GFX90A-NEXT:    s_mul_i32 s23, s13, s9
+; GFX90A-NEXT:    s_add_i32 s11, s22, s11
+; GFX90A-NEXT:    s_mul_i32 s9, s12, s9
 ; GFX90A-NEXT:    s_and_b64 s[4:5], exec, s[4:5]
-; GFX90A-NEXT:    s_add_i32 s13, s13, s23
+; GFX90A-NEXT:    s_add_i32 s11, s11, s23
 ; GFX90A-NEXT:    s_branch .LBB3_5
 ; GFX90A-NEXT:  .LBB3_4: ; %bb58
 ; GFX90A-NEXT:    ; in Loop: Header=BB3_5 Depth=2
@@ -743,7 +743,7 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX90A-NEXT:    ; Parent Loop BB3_2 Depth=1
 ; GFX90A-NEXT:    ; => This Inner Loop Header: Depth=2
 ; GFX90A-NEXT:    s_add_u32 s22, s20, s9
-; GFX90A-NEXT:    s_addc_u32 s23, s21, s13
+; GFX90A-NEXT:    s_addc_u32 s23, s21, s11
 ; GFX90A-NEXT:    global_load_dword v21, v19, s[22:23] offset:16 glc
 ; GFX90A-NEXT:    s_waitcnt vmcnt(0)
 ; GFX90A-NEXT:    global_load_dword v20, v19, s[22:23] offset:20 glc
@@ -780,8 +780,8 @@ define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg
 ; GFX90A-NEXT:    s_cbranch_execnz .LBB3_10
 ; GFX90A-NEXT:  .LBB3_9: ; %bb12
 ; GFX90A-NEXT:    ; in Loop: Header=BB3_2 Depth=1
-; GFX90A-NEXT:    s_add_u32 s10, s10, s8
-; GFX90A-NEXT:    s_addc_u32 s11, s11, 0
+; GFX90A-NEXT:    s_add_u32 s6, s6, s8
+; GFX90A-NEXT:    s_addc_u32 s7, s7, 0
 ; GFX90A-NEXT:    s_add_u32 s14, s14, s16
 ; GFX90A-NEXT:    s_addc_u32 s15, s15, s17
 ; GFX90A-NEXT:    s_branch .LBB3_2
diff --git a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
index a1fc7006b83e4e..efed05114a6345 100644
--- a/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
+++ b/llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll
@@ -2,8 +2,10 @@
 ; RUN: opt -mtriple=amdgpu9.00 -amdgpu-aa -amdgpu-aa-wrapper -amdgpu-annotate-uniform -S < %s | FileCheck %s
 ; RUN: llc -mtriple=amdgpu9.00 -amdgpu-atomic-optimizer-strategy=None < %s | FileCheck -check-prefix=GCN %s
 
-; Check that barrier or fence in between of loads is not considered a clobber
-; for the purpose of converting vector loads into scalar.
+; Check that various operations are correctly considered as clobbering or not
+; clobbering, for the purpose of converting vector loads into scalar.
+; For example, barriers should not be considered clobbering, but global acquire
+; fences should.
 
 @LDS = linkonce_odr hidden local_unnamed_addr addrspace(3) global i32 poison
 
diff --git a/llvm/test/CodeGen/AMDGPU/promote-kernel-arguments.ll b/llvm/test/CodeGen/AMDGPU/promote-kernel-arguments.ll
index 936265345ffb0f..6d8264e558dcd3 100644
--- a/llvm/test/CodeGen/AMDGPU/promote-kernel-arguments.ll
+++ b/llvm/test/CodeGen/AMDGPU/promote-kernel-arguments.ll
@@ -501,11 +501,9 @@ define amdgpu_kernel void @fence_acquire_clobbers_ptr_nest_3(ptr addrspace(1) no
 ; CHECK-NEXT:    [[P1:%.*]] = getelementptr inbounds ptr, ptr addrspace(1) [[ARG:%.*]], i32 [[I]]
 ; CHECK-NEXT:    [[ACQ:%.*]] = load atomic i32, ptr addrspace(1) [[FLAG:%.*]] monotonic, align 4
 ; CHECK-NEXT:    fence acquire
-; CHECK-NEXT:    [[P2:%.*]] = load ptr, ptr addrspace(1) [[P1]], align 8, !amdgpu.noclobber [[META0]]
-; CHECK-NEXT:    [[P2_GLOBAL:%.*]] = addrspacecast ptr [[P2]] to ptr addrspace(1)
-; CHECK-NEXT:    [[P3:%.*]] = load ptr, ptr addrspace(1) [[P2_GLOBAL]], align 8, !amdgpu.noclobber [[META0]]
-; CHECK-NEXT:    [[P3_GLOBAL:%.*]] = addrspacecast ptr [[P3]] to ptr addrspace(1)
-; CHECK-NEXT:    store float 0.000000e+00, ptr addrspace(1) [[P3_GLOBAL]], align 4
+; CHECK-NEXT:    [[P2:%.*]] = load ptr, ptr addrspace(1) [[P1]], align 8
+; CHECK-NEXT:    [[P3:%.*]] = load ptr, ptr [[P2]], align 8
+; CHECK-NEXT:    store float 0.000000e+00, ptr [[P3]], align 4
 ; CHECK-NEXT:    ret void
 ;
 entry:

>From cd29e60f88b9e7eeed2a3b447c3fdd7b9a2b3fcd Mon Sep 17 00:00:00 2001
From: Fabian Ritter <fabian.ritter at amd.com>
Date: Mon, 16 Nov 2026 04:50:10 -0500
Subject: [PATCH 6/6] Formulate the noclobber spec in terms of ordering
 relations

---
 llvm/docs/AMDGPUUsage.md | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/llvm/docs/AMDGPUUsage.md b/llvm/docs/AMDGPUUsage.md
index b5c2d13b24864a..0c1bbfa9a36eee 100644
--- a/llvm/docs/AMDGPUUsage.md
+++ b/llvm/docs/AMDGPUUsage.md
@@ -3307,9 +3307,10 @@ support a 32-byte fetch size so it treats CFS_32B the same as CFS_64B.
 
 ### '`amdgpu.noclobber`' Metadata
 
-If a `load` instruction marked as `!amdgpu.noclobber` may see any write
-other than the initialization of that location at kernel launch according to the
-{ref}`concurrent memory model<amdgpu-location-order>`, it returns `undef`.
+Consider the writes that a `load` instruction L marked as `!amdgpu.noclobber`
+may see according to the {ref}`AMDGPU memory model<amdgpu-location-order>`.
+L returns `undef` if any of these writes does not happen-before the operation
+that dispatches the kernel executing L.
 
 This metadata can allow the AMDGPU compiler backend to emit scalar instead of
 vector load instructions.



More information about the llvm-branch-commits mailing list