[llvm-branch-commits] [llvm] [AMDGPU] Make amdgpu.noclobber respect acquire operations (PR #219991)
via llvm-branch-commits
llvm-branch-commits at lists.llvm.org
Mon Aug 31 07:59:39 PDT 2026
llvmorg-github-actions[bot] wrote:
<!--LLVM PR SUMMARY COMMENT-->
@llvm/pr-subscribers-backend-amdgpu
Author: Fabian Ritter (ritter-x2a)
<details>
<summary>Changes</summary>
AMDGPUAnnotateUniformValues should annotate uniform global loads as
`!amdgpu.noclobber` if the memory they are reading is never clobbered (since
kernel launch) before the load is executed. That allows the backend to use
scalar `s_load_*` instructions instead of vector loads (which would otherwise
be illegal because the scalar cache is not kept coherent with the vector
cache).
So far, loads were annotated if the thread executing the load has not clobbered
the relevant memory location, but changes from concurrent threads that were
acquired via synchronization were ignored (which is unsound).
This patch defines a semantics for `!amdgpu.noclobber` metadata in the
AMDGPUUsage and changes AMDGPUAnnotateUniformValues to respect that semantics.
Now, loads are no longer annotated if they are atomic or if an acquire fence or
load can be executed before them (since they can read from stores in other
threads in these cases).
The patch also fixes an unnecessarily pessimistic case, where atomic stores that
don't alias with the load were considered clobbering.
The noclobber definition uses `undef` instead of `poison` because data races are
currently still defined as `undef`; we need the noclobber definition to use the
same as the data race definition so that it's legal to infer noclobber if the
only way another thread could clobber the memory location would be a data race.
This implementation is overly pessimistic for one-as syncscopes: one-as acquire
operations that don't affect the global addrspace could be ignored for the
noclobber analysis. Improving this is left for a follow-up PR.
For ROCM-29879.
---
Patch is 52.29 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/219991.diff
6 Files Affected:
- (modified) llvm/docs/AMDGPUMemoryModel.md (+2)
- (modified) llvm/docs/AMDGPUUsage.rst (+12)
- (modified) llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp (+7)
- (modified) llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp (+24-12)
- (modified) llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll (+288-280)
- (modified) llvm/test/CodeGen/AMDGPU/noclobber-barrier.ll (+42-51)
``````````diff
diff --git a/llvm/docs/AMDGPUMemoryModel.md b/llvm/docs/AMDGPUMemoryModel.md
index 0a7f023bf19ac..c91200022abc7 100644
--- a/llvm/docs/AMDGPUMemoryModel.md
+++ b/llvm/docs/AMDGPUMemoryModel.md
@@ -339,6 +339,8 @@ and one of the following holds:
Then `Y` makes `W` visible in the intersection `S` of `S1` and `S2`,
and every subscope instance of `S` that includes `Y`.
+(amdgpu-location-order)=
+
### Location Order
A write `W` is *location-ordered* before an access `Y` to the same address
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 40a2820b9e04f..1ee54dccf1e28 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -2490,6 +2490,18 @@ whether to skip the DPP optimization.
; Many expected active lanes: the DPP optimization is not skipped.
%old1 = atomicrmw add ptr addrspace(3) @lds, i32 %val acq_rel, !amdgpu.expected.active.lanes !{i32 32}
+.. _amdgpu_noclobber:
+
+'``amdgpu.noclobber``' Metadata
+-------------------------------------------------
+
+If a ``load`` instruction marked as ``!amdgpu.noclobber`` may see any write
+other than the initialization of that location at kernel launch according to the
+:ref:`concurrent memory model<amdgpu-location-order>`, it returns ``undef``.
+
+This metadata can allow the AMDGPU compiler backend to emit scalar instead of
+vector load instructions.
+
LLVM IR Attributes
==================
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
index ca66ee7721791..855b116598fd5 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUAnnotateUniformValues.cpp
@@ -77,6 +77,13 @@ void AMDGPUAnnotateUniformValues::visitLoadInst(LoadInst &I) {
// for memory operations that are live in to entry points only.
if (!isEntryFunc)
return;
+
+ // If I is atomic, it might see the effects of concurrent clobbering accesses,
+ // so local clobber analysis is not sufficient to ensure that it sees the
+ // initial value.
+ if (I.isAtomic())
+ return;
+
bool GlobalLoad = I.getPointerAddressSpace() == AMDGPUAS::GLOBAL_ADDRESS;
if (GlobalLoad && !AMDGPU::isClobberedInFunction(&I, MSSA, AA))
setNoClobberMetadata(&I);
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
index 781d591118604..c0848bf0433c5 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUMemoryUtils.cpp
@@ -19,6 +19,7 @@
#include "llvm/IR/IntrinsicsAMDGPU.h"
#include "llvm/IR/LLVMContext.h"
#include "llvm/IR/ReplaceConstant.h"
+#include "llvm/Support/AtomicOrdering.h"
#define DEBUG_TYPE "amdgpu-memory-utils"
@@ -367,9 +368,6 @@ void removeFnAttrFromReachable(CallGraph &CG, Function *KernelRoot,
bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA) {
Instruction *DefInst = Def->getMemoryInst();
- if (isa<FenceInst>(DefInst))
- return false;
-
if (const IntrinsicInst *II = dyn_cast<IntrinsicInst>(DefInst)) {
switch (II->getIntrinsicID()) {
case Intrinsic::amdgcn_s_barrier:
@@ -393,15 +391,29 @@ bool isReallyAClobber(const Value *Ptr, MemoryDef *Def, AAResults *AA) {
}
}
- // Ignore atomics not aliasing with the original load, any atomic is a
- // universal MemoryDef from MSSA's point of view too, just like a fence.
- const auto checkNoAlias = [AA, Ptr](auto I) -> bool {
- return I && AA->isNoAlias(I->getPointerOperand(), Ptr);
+ // Ignore non-acquire atomics not aliasing with the original load, any atomic
+ // is a universal MemoryDef from MSSA's point of view too, just like a fence.
+ // Acquire (or stronger) fences/atomics act as clobbers because they can bring
+ // in effects from other threads.
+ const auto mayAlias = [AA, Ptr](auto I) -> bool {
+ return !AA->isNoAlias(I->getPointerOperand(), Ptr);
};
- if (checkNoAlias(dyn_cast<AtomicCmpXchgInst>(DefInst)) ||
- checkNoAlias(dyn_cast<AtomicRMWInst>(DefInst)))
- return false;
+ if (const auto *F = dyn_cast<FenceInst>(DefInst))
+ return isAcquireOrStronger(F->getOrdering());
+
+ if (const auto *I = dyn_cast<AtomicRMWInst>(DefInst))
+ return isAcquireOrStronger(I->getOrdering()) || mayAlias(I);
+
+ if (const auto *I = dyn_cast<AtomicCmpXchgInst>(DefInst))
+ return isAcquireOrStronger(I->getMergedOrdering()) || mayAlias(I);
+
+ if (const auto *I = dyn_cast<LoadInst>(DefInst))
+ return isAcquireOrStronger(I->getOrdering());
+
+ // (Atomic) stores that don't alias do not clobber.
+ if (const auto *I = dyn_cast<StoreInst>(DefInst))
+ return mayAlias(I);
return true;
}
@@ -423,8 +435,8 @@ bool isClobberedInFunction(const LoadInst *Load, MemorySSA *MSSA,
// case add all Defs to WorkList and continue going up and checking all
// the definitions of this memory location until the root. When all the
// defs are exhausted and came to the entry state we have no clobber.
- // Along the scan ignore barriers and fences which are considered clobbers
- // by the MemorySSA, but not really writing anything into the memory.
+ // Along the scan ignore barriers which are considered clobbers by the
+ // MemorySSA, but not really writing anything into the memory.
while (!WorkList.empty()) {
MemoryAccess *MA = WorkList.pop_back_val();
if (!Visited.insert(MA).second)
diff --git a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
index 41c3e0a07f5f5..960aa6932c9ed 100644
--- a/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
+++ b/llvm/test/CodeGen/AMDGPU/agpr-copy-no-free-registers.ll
@@ -510,338 +510,346 @@ define void @v32_asm_def_use(float %v0, float %v1) #4 {
define amdgpu_kernel void @introduced_copy_to_sgpr(i64 %arg, i32 %arg1, i32 %arg2, i64 %arg3, <2 x half> %arg4, <2 x half> %arg5) #3 {
; GFX908-LABEL: introduced_copy_to_sgpr:
; GFX908: ; %bb.0: ; %bb
-; GFX908-NEXT: global_load_ushort v0, v[0:1], off glc
-; GFX908-NEXT: s_load_dwordx4 s[0:3], s[8:9], 0x0
-; GFX908-NEXT: s_load_dwordx2 s[4:5], s[8:9], 0x10
-; GFX908-NEXT: s_load_dword s7, s[8:9], 0x18
-; GFX908-NEXT: s_mov_b32 s6, 0
-; GFX908-NEXT: s_mov_b32 s9, s6
+; GFX908-NEXT: global_load_ushort v16, v[0:1], off glc
+; GFX908-NEXT: s_load_dwordx4 s[4:7], s[8:9], 0x0
+; GFX908-NEXT: s_load_dwordx2 s[2:3], s[8:9], 0x10
+; GFX908-NEXT: s_load_dword s0, s[8:9], 0x18
+; GFX908-NEXT: s_mov_b32 s10, 0
+; GFX908-NEXT: s_mov_b32 s9, s10
; GFX908-NEXT: s_waitcnt lgkmcnt(0)
-; GFX908-NEXT: v_cvt_f32_u32_e32 v1, s3
-; GFX908-NEXT: s_sub_i32 s8, 0, s3
-; GFX908-NEXT: v_cvt_f32_f16_e32 v12, s7
-; GFX908-NEXT: s_mov_b64 s[16:17], 0
-; GFX908-NEXT: v_rcp_f32_e32 v1, v1
-; GFX908-NEXT: v_mov_b32_e32 v14, 0
-; GFX908-NEXT: v_mul_f32_e32 v1, 0x4f7ffffe, v1
-; GFX908-NEXT: v_cvt_u32_f32_e32 v1, v1
-; GFX908-NEXT: v_readfirstlane_b32 s10, v1
-; GFX908-NEXT: s_mul_i32 s8, s8, s10
-; GFX908-NEXT: s_mul_hi_u32 s8, s10, s8
-; GFX908-NEXT: s_add_i32 s10, s10, s8
-; GFX908-NEXT: s_mul_hi_u32 s8, s2, s10
-; GFX908-NEXT: s_mul_i32 s10, s8, s3
-; GFX908-NEXT: s_sub_i32 s2, s2, s10
-; GFX908-NEXT: s_add_i32 s11, s8, 1
-; GFX908-NEXT: s_sub_i32 s10, s2, s3
-; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: v_readfirstlane_b32 s12, v0
-; GFX908-NEXT: s_and_b32 s30, s12, 0xffff
-; GFX908-NEXT: s_cmp_ge_u32 s2, s3
-; GFX908-NEXT: s_cselect_b32 s8, s11, s8
-; GFX908-NEXT: s_cselect_b32 s2, s10, s2
-; GFX908-NEXT: s_add_i32 s10, s8, 1
-; GFX908-NEXT: s_cmp_ge_u32 s2, s3
-; GFX908-NEXT: s_cselect_b32 s8, s10, s8
-; GFX908-NEXT: s_lshr_b32 s7, s7, 16
-; GFX908-NEXT: v_cvt_f32_f16_e32 v13, s7
-; GFX908-NEXT: s_mul_i32 s12, s1, s30
-; GFX908-NEXT: s_mul_hi_u32 s13, s0, s30
-; GFX908-NEXT: s_mul_i32 s14, s0, s30
-; GFX908-NEXT: s_add_i32 s15, s13, s12
-; GFX908-NEXT: s_lshl_b64 s[2:3], s[0:1], 5
-; GFX908-NEXT: s_lshl_b64 s[10:11], s[4:5], 5
-; GFX908-NEXT: s_lshl_b64 s[12:13], s[8:9], 5
-; GFX908-NEXT: s_lshl_b64 s[14:15], s[14:15], 5
+; GFX908-NEXT: v_cvt_f32_u32_e32 v0, s7
+; GFX908-NEXT: s_sub_i32 s1, 0, s7
+; GFX908-NEXT: v_cvt_f32_f16_e32 v18, s0
+; GFX908-NEXT: v_mov_b32_e32 v17, 0
+; GFX908-NEXT: v_rcp_f32_e32 v0, v0
+; GFX908-NEXT: v_mov_b32_e32 v1, 0
+; GFX908-NEXT: v_mul_f32_e32 v0, 0x4f7ffffe, v0
+; GFX908-NEXT: v_cvt_u32_f32_e32 v0, v0
+; GFX908-NEXT: v_readfirstlane_b32 s8, v0
+; GFX908-NEXT: s_mul_i32 s1, s1, s8
+; GFX908-NEXT: s_mul_hi_u32 s1, s8, s1
+; GFX908-NEXT: s_add_i32 s8, s8, s1
+; GFX908-NEXT: s_mul_hi_u32 s1, s6, s8
+; GFX908-NEXT: s_mul_i32 s8, s1, s7
+; GFX908-NEXT: s_sub_i32 s6, s6, s8
+; GFX908-NEXT: s_add_i32 s11, s1, 1
+; GFX908-NEXT: s_sub_i32 s8, s6, s7
+; GFX908-NEXT: s_cmp_ge_u32 s6, s7
+; GFX908-NEXT: s_cselect_b32 s1, s11, s1
+; GFX908-NEXT: s_cselect_b32 s6, s8, s6
+; GFX908-NEXT: s_add_i32 s8, s1, 1
+; GFX908-NEXT: s_cmp_ge_u32 s6, s7
+; GFX908-NEXT: s_cselect_b32 s8, s8, s1
+; GFX908-NEXT: s_lshr_b32 s11, s0, 16
+; GFX908-NEXT: s_lshl_b64 s[14:15], s[8:9], 5
+; GFX908-NEXT: v_cvt_f32_f16_e32 v19, s11
+; GFX908-NEXT: s_lshl_b64 s[6:7], s[4:5], 5
+; GFX908-NEXT: s_lshl_b64 s[12:13], s[2:3], 5
; GFX908-NEXT: s_and_b64 s[0:1], exec, s[0:1]
+; GFX908-NEXT: v_mov_b32_e32 v0, 0
+; GFX908-NEXT: s_waitcnt vmcnt(0)
+; GFX908-NEXT: v_readfirstlane_b32 s9, v16
+; GFX908-NEXT: s_and_b32 s9, 0xffff, s9
+; GFX908-NEXT: s_mul_i32 s5, s5, s9
+; GFX908-NEXT: s_mul_hi_u32 s11, s4, s9
+; GFX908-NEXT: s_mul_i32 s4, s4, s9
+; GFX908-NEXT: s_add_i32 s5, s11, s5
+; GFX908-NEXT: s_lshl_b64 s[4:5], s[4:5], 5
; GFX908-NEXT: s_branch .LBB3_2
; GFX908-NEXT: .LBB3_1: ; %Flow20
; GFX908-NEXT: ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT: s_and_b64 s[18:19], s[18:19], exec
-; GFX908-NEXT: s_cselect_b32 s7, 1, 0
-; GFX908-NEXT: s_cmp_lg_u32 s7, 1
-; GFX908-NEXT: s_cbranch_scc0 .LBB3_13
+; GFX908-NEXT: s_and_b64 s[16:17], s[16:17], exec
+; GFX908-NEXT: s_cselect_b32 s9, 1, 0
+; GFX908-NEXT: s_cmp_lg_u32 s9, 1
+; GFX908-NEXT: s_cbranch_scc0 .LBB3_14
; GFX908-NEXT: .LBB3_2: ; %bb9
; GFX908-NEXT: ; =>This Loop Header: Depth=1
-; GFX908-NEXT: ; Child Loop BB3_5 Depth 2
-; GFX908-NEXT: s_mov_b64 s[20:21], -1
+; GFX908-NEXT: ; Child Loop BB3_6 Depth 2
+; GFX908-NEXT: s_mov_b64 s[18:19], -1
; GFX908-NEXT: s_mov_b64 vcc, s[0:1]
-; GFX908-NEXT: s_cbranch_vccz .LBB3_11
+; GFX908-NEXT: s_cbranch_vccz .LBB3_12
; GFX908-NEXT: ; %bb.3: ; %bb14
; GFX908-NEXT: ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT: s_load_dwordx2 s[18:19], s[16:17], 0x0
-; GFX908-NEXT: s_cmp_lt_i32 s5, 0
+; GFX908-NEXT: global_load_dwordx2 v[2:3], v[0:1], off
+; GFX908-NEXT: s_cmp_lt_i32 s3, 0
+; GFX908-NEXT: s_mov_b32 s11, s10
+; GFX908-NEXT: s_cselect_b64 s[16:17], -1, 0
+; GFX908-NEXT: s_cmp_gt_i32 s3, -1
+; GFX908-NEXT: v_mov_b32_e32 v4, s10
+; GFX908-NEXT: v_mov_b32_e32 v5, s11
; GFX908-NEXT: s_cselect_b64 s[20:21], -1, 0
-; GFX908-NEXT: s_cmp_gt_i32 s5, -1
-; GFX908-NEXT: s_cselect_b64 s[22:23], -1, 0
-; GFX908-NEXT: s_waitcnt lgkmcnt(0)
-; GFX908-NEXT: s_add_u32 s7, s18, 1
-; GFX908-NEXT: s_addc_u32 s9, s19, 0
-; GFX908-NEXT: s_mul_i32 s9, s2, s9
-; GFX908-NEXT: s_mul_hi_u32 s24, s2, s7
-; GFX908-NEXT: s_add_i32 s9, s24, s9
-; GFX908-NEXT: s_mul_i32 s24, s3, s7
-; GFX908-NEXT: s_mul_i32 s31, s2, s7
-; GFX908-NEXT: s_mov_b32 s7, s6
-; GFX908-NEXT: s_add_i32 s9, s9, s24
-; GFX908-NEXT: v_mov_b32_e32 v0, s6
-; GFX908-NEXT: v_mov_b32_e32 v1, s7
-; GFX908-NEXT: s_mov_b64 s[24:25], s[10:11]
-; GFX908-NEXT: v_mov_b32_e32 v2, s6
-; GFX908-NEXT: v_mov_b32_e32 v3, s7
-; GFX908-NEXT: v_mov_b32_e32 v4, s6
-; GFX908-NEXT: v_mov_b32_e32 v5, s7
-; GFX908-NEXT: v_mov_b32_e32 v6, s6
-; GFX908-NEXT: v_mov_b32_e32 v7, s7
-; GFX908-NEXT: s_branch .LBB3_5
-; GFX908-NEXT: .LBB3_4: ; %Flow18
-; GFX908-NEXT: ; in Loop: Header=BB3_5 Depth=2
-; GFX908-NEXT: s_and_b64 s[28:29], s[28:29], exec
-; GFX908-NEXT: s_cselect_b32 s7, 1, 0
-; GFX908-NEXT: s_cmp_lg_u32 s7, 1
-; GFX908-NEXT: s_cbranch_scc0 .LBB3_10
-; GFX908-NEXT: .LBB3_5: ; %bb16
+; GFX908-NEXT: v_mov_b32_e32 v6, s10
+; GFX908-NEXT: v_mov_b32_e32 v7, s11
+; GFX908-NEXT: v_mov_b32_e32 v8, s10
+; GFX908-NEXT: v_mov_b32_e32 v9, s11
+; GFX908-NEXT: s_mov_b64 s[18:19], s[12:13]
+; GFX908-NEXT: v_mov_b32_e32 v11, v5
+; GFX908-NEXT: v_mov_b32_e32 v10, v4
+; GFX908-NEXT: s_waitcnt vmcnt(0)
+; GFX908-NEXT: v_readfirstlane_b32 s9, v2
+; GFX908-NEXT: v_readfirstlane_b32 s11, v3
+; GFX908-NEXT: s_add_u32 s9, s9, 1
+; GFX908-NEXT: s_addc_u32 s11, s11, 0
+; GFX908-NEXT: s_mul_hi_u32 s22, s6, s9
+; GFX908-NEXT: s_mul_i32 s11, s6, s11
+; GFX908-NEXT: s_mul_i32 s23, s7, s9
+; GFX908-NEXT: s_add_i32 s11, s22, s11
+; GFX908-NEXT: s_mul_i32 s9, s6, s9
+; GFX908-NEXT: s_add_i32 s11, s11, s23
+; GFX908-NEXT: s_branch .LBB3_6
+; GFX908-NEXT: .LBB3_4: ; %bb58
+; GFX908-NEXT: ; in Loop: Header=BB3_6 Depth=2
+; GFX908-NEXT: v_add_co_u32_sdwa v2, vcc, v2, v16 dst_sel:DWORD dst_unused:UNUSED_PAD src0_sel:DWORD src1_sel:WORD_0
+; GFX908-NEXT: v_addc_co_u32_e32 v3, vcc, 0, v3, vcc
+; GFX908-NEXT: s_add_u32 s18, s18, s4
+; GFX908-NEXT: s_addc_u32 s19, s19, s5
+; GFX908-NEXT: s_mov_b64 s[22:23], 0
+; GFX908-NEXT: v_cmp_lt_i32_e64 s[24:25], -1, v3
+; GFX908-NEXT: .LBB3_5: ; %Flow18
+; GFX908-NEXT: ; in Loop: Header=BB3_6 Depth=2
+; GFX908-NEXT: s_and_b64 s[24:25], s[24:25], exec
+; GFX908-NEXT: s_cselect_b32 s24, 1, 0
+; GFX908-NEXT: s_cmp_lg_u32 s24, 1
+; GFX908-NEXT: s_cbranch_scc0 .LBB3_11
+; GFX908-NEXT: .LBB3_6: ; %bb16
; GFX908-NEXT: ; Parent Loop BB3_2 Depth=1
; GFX908-NEXT: ; => This Inner Loop Header: Depth=2
-; GFX908-NEXT: s_add_u32 s26, s24, s31
-; GFX908-NEXT: s_addc_u32 s27, s25, s9
-; GFX908-NEXT: global_load_dword v16, v14, s[26:27] offset:16 glc
+; GFX908-NEXT: s_add_u32 s22, s18, s9
+; GFX908-NEXT: s_addc_u32 s23, s19, s11
+; GFX908-NEXT: global_load_dword v21, v17, s[22:23] offset:16 glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: global_load_dword v15, v14, s[26:27] offset:20 glc
+; GFX908-NEXT: global_load_dword v20, v17, s[22:23] offset:20 glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: global_load_dword v8, v14, s[26:27] offset:24 glc
+; GFX908-NEXT: global_load_dword v12, v17, s[22:23] offset:24 glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: global_load_dword v8, v14, s[26:27] offset:28 glc
+; GFX908-NEXT: global_load_dword v12, v17, s[22:23] offset:28 glc
; GFX908-NEXT: s_waitcnt vmcnt(0)
-; GFX908-NEXT: ds_read_b64 v[8:9], v14
-; GFX908-NEXT: ds_read_b64 v[10:11], v0
-; GFX908-NEXT: ; kill: killed $sgpr26 killed $sgpr27
-; GFX908-NEXT: s_and_b64 s[26:27], s[22:23], exec
-; GFX908-NEXT: s_cselect_b32 s7, 1, 0
-; GFX908-NEXT: s_cmp_lg_u32 s7, 1
+; GFX908-NEXT: ds_read_b64 v[12:13], v17
+; GFX908-NEXT: ds_read_b64 v[14:15], v0
+; GFX908-NEXT: s_and_b64 s[22:23], s[20:21], exec
+; GFX908-NEXT: s_cselect_b32 s22, 1, 0
+; GFX908-NEXT: s_cmp_lg_u32 s22, 1
; GFX908-NEXT: s_waitcnt lgkmcnt(0)
-; GFX908-NEXT: s_cbranch_scc1 .LBB3_7
-; GFX908-NEXT: ; %bb.6: ; %bb51
-; GFX908-NEXT: ; in Loop: Header=BB3_5 Depth=2
-; GFX908-NEXT: v_add_f32_e32 v17, v12, v8
-; GFX908-NEXT: v_add_f32_e32 v18, v13, v9
-; GFX908-NEXT: v_add_f32_e32 v19, 0, v8
-; GFX908-NEXT: v_add_f32_e32 v20, 0, v9
-; GFX908-NEXT: v_fma_mix_f32 v10, v16, 1.0, v10 op_sel_hi:[1,1,0]
-; GFX908-NEXT: v_fma_mix_f32 v11, v16, 1.0, v11 op_sel:[1,0,0] op_sel_hi:[1,1,0]
-; GFX908-NEXT: v_fma_mix_f32 v8, v15, 1.0, v8 op_sel_hi:[1,1,0]
-; GFX908-NEXT: v_fma_mix_f32 v9, v15, 1.0, v9 op_sel:[1,0,0] op_sel_hi:[1,1,0]
-; GFX908-NEXT: v_add_f32_e32 v1, v1, v18
-; GFX908-NEXT: v_add_f32_e32 v0, v0, v17
-; GFX908-NEXT: v_add_f32_e32 v3, v3, v20
-; GFX908-NEXT: v_add_f32_e32 v2, v2, v19
-; GFX908-NEXT: v_add_f32_e32 v5, v5, v11
-; GFX908-NEXT: v_add_f32_e32 v4, v4, v10
-; GFX908-NEXT: v_add_f32_e32 v7, v7, v9
-; GFX908-NEXT: v_add_f32_e32 v6, v6, v8
-; GFX908-NEXT: s_mov_b64 s[28:29], -1
-; GFX908-NEXT: s_branch .LBB3_8
-; GFX908-NEXT: .LBB3_7: ; in Loop: Header=BB3_5 Depth=2
-; GFX908-NEXT: s_mov_b64 s[28:29], s[20:21]
-; GFX908-NEXT: .LBB3_8: ; %Flow
-; GFX908-NEXT: ; in Loop: Header=BB3_5 Depth=2
-; GFX908-NEXT: s_and_b64 s[28:29], s[28:29], exec
-; GFX908-NEXT: s_cselect_b32 s7, 1, 0
-; GFX908-NEXT: s_mov_b64 s[26:27], -1
-; GFX908-NEXT: s_cmp_lg_u32 s7, 1
-; GFX908-NEXT: s_mov_b64 s[28:29], -1
-; GFX908-NEXT: s_cbranch_scc1 .LBB3_4
-; GFX908-NEXT: ; %bb.9: ; %bb58
-; GFX908-NEXT: ; in Loop: Header=BB3_5 Depth=2
-; GFX908-NEXT: s_add_u32 s18, s18, s30
-; GFX908-NEXT: s_addc_u32 s19, s19, 0
-; GFX908-NEXT: s_add_u32 s24, s24, s14
-; GFX908-NEXT: s_addc_u32 s25, s25, s15
-; GFX908-NEXT: s_cmp_gt_i32 s19, -1
-; GFX908-NEXT: s_mov_b64 s[26:27], 0
-; GFX908-NEXT: s_cselect_b64 s[28:29], -1, 0
-; GFX908-NEXT: s_branch .LBB3_4
-; GFX908-NEXT: .LBB3_10: ; %loop.exit.guard
+; GFX908-NEXT: s_cbranch_scc1 .LBB3_8
+; GFX908-NEXT: ; %bb.7: ; %bb51
+; GFX908-NEXT: ; in Loop: Header=BB3_6 Depth=2
+; GFX908-NEXT: v_add_f32_e32 v22, v18, v12
+; GFX908-NEXT: v_add_f32_e32 v23, v19, v13
+; GFX908-NEXT: v_add_f32_e32 v24, 0, v12
+; GFX908-NEXT: v_add_f32_e32 v25, 0, v13
+; GFX908-NEXT: v_fma_mix_f32 v14, v21, 1.0, v14 op_sel_hi:[1,1,0]
+; GFX908-NEXT: v_fma_mix_f32 v15, v21, 1.0, v15 op_sel:[1,0,0] op_sel_hi:[1,1,0]
+; GFX908-NEXT: v_fma_mix_f32 v12, v20, 1.0, v12 op_sel_hi:[1,1,0]
+; GFX908-NEXT: v_fma_mix_f32 v13, v20, 1.0, v13 op_sel:[1,0,0] op_sel_hi:[1,1,0]
+; GFX908-NEXT: v_add_f32_e32 v5, v5, v23
+; GFX908-NEXT: v_add_f32_e32 v4, v4, v22
+; GFX908-NEXT: v_add_f32_e32 v7, v7, v25
+; GFX908-NEXT: v_add_f32_e32 v6, v6, v24
+; GFX908-NEXT: v_add_f32_e32 v9, v9, v15
+; GFX908-NEXT: v_add_f32_e32 v8, v8, v14
+; GFX908-NEXT: v_add_f32_e32 v11, v11, v13
+; GFX908-NEXT: v_add_f32_e32 v10, v10, v12
+; GFX908-NEXT: s_mov_b64 s[22:23], -1
+; GFX908-NEXT: s_branch .LBB3_9
+; GFX908-NEXT: .LBB3_8: ; in Loop: Header=BB3_6 Depth=2
+; GFX908-NEXT: s_mov_b64 s[22:23], s[16:17]
+; GFX908-NEXT: .LBB3_9: ; %Flow
+; GFX908-NEXT: ; in Loop: Header=BB3_6 Depth=2
+; GFX908-NEXT: s_and_b64 s[22:23], s[22:23], exec
+; GFX908-NEXT: s_cselect_b32 s22, 1, 0
+; GFX908-NEXT: s_cmp_lg_u32 s22, 1
+; GFX908-NEXT: s_cbranch_scc0 .LBB3_4
+; GFX908-NEXT: ; %bb.10: ; in Loop: Header=BB3_6 Depth=2
+; GFX908-NEXT: s_mov_b64 s[22:23], -1
+; GFX908-NEXT: ; implicit-def: $vgpr2_vgpr3
+; GFX908-NEXT: ; implicit-def: $sgpr18_sgpr19
+; GFX908-NEXT: s_mov_b64 s[24:25], -1
+; GFX908-NEXT: s_branch .LBB3_5
+; GFX908-NEXT: .LBB3_11: ; %loop.exit.guard
; GFX908-NEXT: ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT: s_xor_b64 s[20:21], s[26:27], -1
-; GFX908-NEXT: .LBB3_11: ; %Flow19
+; GFX908-NEXT: s_xor_b64 s[18:19], s[22:23], -1
+; GFX908-NEXT: .LBB3_12: ; %Flow19
; GFX908-NEXT: ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT: s_mov_b64 s[18:19], -1
-; GFX908-NEXT: s_and_b64 vcc, exec, s[20:21]
+; GFX908-NEXT: s_mov_b64 s[16:17], -1
+; GFX908-NEXT: s_and_b64 vcc, exec, s[18:19]
; GFX908-NEXT: s_cbranch_vccz .LBB3_1
-; GFX908-NEXT: ; %bb.12: ; %bb12
+; GFX908-NEXT: ; %bb.13: ; %bb12
; GFX908-NEXT: ; in Loop: Header=BB3_2 Depth=1
-; GFX908-NEXT: s_add_u32 s4, s4, s8
-; GFX908-NEXT: s_addc_u32 s5, s5, 0
-; GFX908-NEXT: s_add_u32 s10, s10, s12
-; GFX908-NEXT: s_addc_u32 s11, s11, s13
-; GFX908-NEXT: s_mov_b64 s[18:19], 0
+; GFX908-NEXT: s_add_u32 s2, s2, s8
+; GFX908-NEXT: s_addc_u32 s3, s3, 0
+; GFX908-NEXT: s_add_u32 s12, s12, s14
+; GFX908-NEXT: s_addc_u32 s13, s13, s15
+; GFX908-NEXT: s_mov_b64 s[16:17], 0
; GFX908-NEXT: s_branch .LBB3_1
-; GFX908-NEXT: .LBB3_13: ; %DummyReturnBlock
+; GFX908-NEXT: .LBB3_14: ; %DummyReturnBlock
; GFX908-NEXT: s_endpgm
;
; GFX90A-LABEL: introduced_copy_to_sgpr:
; GFX90A: ; %bb.0: ; %bb
-; GFX90A-NEXT:...
[truncated]
``````````
</details>
https://github.com/llvm/llvm-project/pull/219991
More information about the llvm-branch-commits
mailing list