[llvm] [AMDGPU] Make globally-addressable-scratch opt-in (PR #189555)
Pierre van Houtryve via llvm-commits
llvm-commits at lists.llvm.org
Wed Sep 9 03:07:46 PDT 2026
https://github.com/Pierre-vh updated https://github.com/llvm/llvm-project/pull/189555
>From c2c8fcc4770053f94ae84f6ab689a44a48af5295 Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Mon, 30 Mar 2026 11:55:36 +0200
Subject: [PATCH 1/2] [AMDGPU] Make globally-addressable-scratch opt-in
This feature is meant to be opt-in for more advanced users, not default-enabled.
It may reduce performance otherwise as we can't assume private AS is thread-local
when it is enabled.
Make it controllable via a `-amdgpu-globally-addressable-scratch` CL option and disable it by default.
---
llvm/docs/AMDGPUUsage.rst | 22 +-
llvm/lib/Target/AMDGPU/AMDGPU.td | 11 +-
.../lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp | 8 +-
.../AMDGPU/AMDGPUTargetTransformInfo.cpp | 2 +-
.../AMDGPU/AsmParser/AMDGPUAsmParser.cpp | 6 +-
llvm/lib/Target/AMDGPU/GCNSubtarget.cpp | 9 +
llvm/lib/Target/AMDGPU/GCNSubtarget.h | 2 +
llvm/lib/Target/AMDGPU/SIISelLowering.cpp | 14 +-
llvm/lib/Target/AMDGPU/SIInstrInfo.cpp | 2 +-
llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp | 2 +-
.../AMDGPU/memory-legalizer-private-agent.ll | 3 +-
.../memory-legalizer-private-cluster.ll | 3 +-
.../memory-legalizer-private-lastuse.ll | 6 +-
.../memory-legalizer-private-nontemporal.ll | 3 +-
.../memory-legalizer-private-singlethread.ll | 3 +-
.../AMDGPU/memory-legalizer-private-system.ll | 9939 ++++++++++-------
.../memory-legalizer-private-volatile.ll | 6 +-
.../memory-legalizer-private-wavefront.ll | 3 +-
.../memory-legalizer-private-workgroup.ll | 3 +-
.../AMDGPU/expand-atomic-private-gas.ll | 12 +-
20 files changed, 5731 insertions(+), 4328 deletions(-)
diff --git a/llvm/docs/AMDGPUUsage.rst b/llvm/docs/AMDGPUUsage.rst
index 8012631273863..1521f7359fb1a 100644
--- a/llvm/docs/AMDGPUUsage.rst
+++ b/llvm/docs/AMDGPUUsage.rst
@@ -702,6 +702,9 @@ Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following
=========== =============== =============== ===== ================= =============== =============== ======================
+:sup:`1`: Globally addressable scratch (GAS) is not enabled by default, but can be enabled using ``-amdgpu-globally-addressable-scratch``.
+There is currently no front-end support for GAS, and thus back-end support is limited.
+
Generic processors allow execution of a single code object on any of the processors that
it supports. Such code objects may not perform as well as those for
the non-generic processors.
@@ -1242,8 +1245,8 @@ supported for the ``amdgcn`` target.
access is not supported except by flat and scratch instructions in
GFX9-GFX11.
- On targets without "Globally Accessible Scratch" (introduced in GFX125x), code that
- manipulates the stack values in other lanes of a wavefront, such as by
+ If globally addressable scratch is not supported by the target, or is not enabled,
+ code that manipulates the stack values in other lanes of a wavefront, such as by
``addrspacecast``-ing stack pointers to generic ones and taking offsets that reach other
lanes or by explicitly constructing the scratch buffer descriptor, triggers undefined
behavior when it modifies the scratch values of other lanes. The compiler may assume
@@ -7505,9 +7508,14 @@ termed vector memory operations.
are :ref:`LDS DMA operations<amdgpu-dma-operations>`.
Private address space uses ``buffer_load/store`` using the scratch V#
-(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11). Since only a single thread
-is accessing the memory, atomic memory orderings are not meaningful, and all
-accesses are treated as non-atomic.
+(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11).
+
+* On targets where globally addressable scratch is not supported, or not enabled,
+ Only a single thread can access private memory. Thus, atomic memory orderings
+ are not meaningful, and all private accesses are treated as non-atomic.
+* On targets where globally addressable scratch is supported **and** enabled, private
+ atomic accesses are converted into atomics in the generic (``flat``) address space.
+ All properties of the atomic (atomic ordering, volatility, alignment, etc.) are preserved.
Constant address space uses ``buffer/global_load`` instructions (or equivalent
scalar memory instructions). Since the constant address space contents do not
@@ -17726,8 +17734,8 @@ For GFX125x:
This section is currently incomplete as work on the compiler is still ongoing.
The following is a non-exhaustive list of unimplemented/undocumented features:
- non-volatile bit code sequences, globally accessing scratch atomics,
- multicast loads, barriers (including split barriers) and cooperative atomics.
+ non-volatile bit code sequences, multicast loads, barriers (including split barriers)
+ and cooperative atomics.
Scalar operations memory model needs more elaboration as well.
* Vector memory operations are performed as wavefront wide operations, with the
diff --git a/llvm/lib/Target/AMDGPU/AMDGPU.td b/llvm/lib/Target/AMDGPU/AMDGPU.td
index 29ec1ad5a419b..b967119d1d6e5 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPU.td
+++ b/llvm/lib/Target/AMDGPU/AMDGPU.td
@@ -1512,9 +1512,10 @@ defm XF32Insts : AMDGPUSubtargetFeature<"xf32-insts",
"v_mfma_f32_16x16x8_xf32 and v_mfma_f32_32x32x4_xf32"
>;
-defm GloballyAddressableScratch : AMDGPUSubtargetFeature<"globally-addressable-scratch",
- "FLAT instructions can access scratch memory for any thread in any wave",
- /*GenPredicate=*/0
+def FeatureGloballyAddressableScratchSupport : SubtargetFeature<"globally-addressable-scratch-support",
+ "HasGloballyAddressableScratchSupport",
+ "true",
+ "Hardware supports Globally Addressable Scratch"
>;
// Enable the use of SCRATCH_STORE/LOAD_BLOCK instructions for saving and
@@ -2437,7 +2438,7 @@ def FeatureISAVersion12_50_Common : FeatureSet<
FeatureFlatBufferGlobalAtomicFaddF64Inst,
FeatureMemoryAtomicFAddF32DenormalSupport,
FeatureEmulatedSystemScopeAtomics,
- FeatureGloballyAddressableScratch,
+ FeatureGloballyAddressableScratchSupport,
FeatureKernargPreload,
FeatureVmemPrefInsts,
FeatureLshlAddU64Inst,
@@ -2619,7 +2620,7 @@ def FeatureISAVersion13 : FeatureSet<
FeatureAtomicFMinFMaxF64GlobalInsts,
FeatureAtomicFMinFMaxF64FlatInsts,
FeatureFmaMixBF16Insts,
- FeatureGloballyAddressableScratch,
+ FeatureGloballyAddressableScratchSupport,
FeatureCvtPkF16F32Inst,
FeatureCvtSrPkBF16F32Inst,
FeatureF16BF16ToFP6BF6ConversionScaleInsts,
diff --git a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
index 6d722311d6d5d..3a779b1b4b4e2 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPULegalizerInfo.cpp
@@ -2458,7 +2458,7 @@ Register AMDGPULegalizerInfo::getBaseSegmentAperture(
const unsigned ApertureRegNo =
IsLDS ? AMDGPU::SRC_SHARED_BASE : AMDGPU::SRC_PRIVATE_BASE;
assert((ApertureRegNo != AMDGPU::SRC_PRIVATE_BASE ||
- !ST.hasGloballyAddressableScratch()) &&
+ !ST.hasGloballyAddressableScratchSupport()) &&
"Cannot use src_private_base with globally addressable scratch!");
Register Dst = MRI.createGenericVirtualRegister(I64);
MRI.setRegClass(Dst, &AMDGPU::SReg_64RegClass);
@@ -2583,7 +2583,7 @@ bool AMDGPULegalizerInfo::legalizeAddrSpaceCast(
DestAS == AMDGPUAS::PRIVATE_ADDRESS)) {
auto castFlatToLocalOrPrivate = [&](const DstOp &Dst) -> Register {
if (DestAS == AMDGPUAS::PRIVATE_ADDRESS &&
- ST.hasGloballyAddressableScratch()) {
+ ST.hasGloballyAddressableScratchSupport()) {
// flat -> private with globally addressable scratch: subtract
// src_flat_scratch_base_lo.
Register SrcLo = B.buildExtract(I32, Src, 0).getReg(0);
@@ -2630,7 +2630,7 @@ bool AMDGPULegalizerInfo::legalizeAddrSpaceCast(
Register SrcAsInt = B.buildPtrToInt(I32, Src).getReg(0);
if (SrcAS == AMDGPUAS::PRIVATE_ADDRESS &&
- ST.hasGloballyAddressableScratch()) {
+ ST.hasGloballyAddressableScratchSupport()) {
// For wave32: Addr = (TID[4:0] << 52) + FLAT_SCRATCH_BASE + privateAddr
// For wave64: Addr = (TID[5:0] << 51) + FLAT_SCRATCH_BASE + privateAddr
Register AllOnes = B.buildConstant(I32, -1).getReg(0);
@@ -6531,7 +6531,7 @@ bool AMDGPULegalizerInfo::legalizeIsAddrSpace(MachineInstr &MI,
Register Hi32 = Unmerge.getReg(1);
if (AddrSpace == AMDGPUAS::PRIVATE_ADDRESS &&
- ST.hasGloballyAddressableScratch()) {
+ ST.hasGloballyAddressableScratchSupport()) {
Register FlatScratchBaseHi =
B.buildInstr(AMDGPU::S_MOV_B32, {I32},
{Register(AMDGPU::SRC_FLAT_SCRATCH_BASE_HI)})
diff --git a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
index a7556278b7e0d..beffd3c9b476b 100644
--- a/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/AMDGPUTargetTransformInfo.cpp
@@ -1229,7 +1229,7 @@ bool GCNTTIImpl::isSourceOfDivergence(const Value *V) const {
if (auto *CastI = dyn_cast<AddrSpaceCastInst>(V)) {
return CastI->getSrcAddressSpace() == AMDGPUAS::PRIVATE_ADDRESS &&
CastI->getDestAddressSpace() == AMDGPUAS::FLAT_ADDRESS &&
- ST->hasGloballyAddressableScratch();
+ ST->hasGloballyAddressableScratchSupport();
}
return false;
diff --git a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
index 4db11eace7313..f715c9f20a038 100644
--- a/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
+++ b/llvm/lib/Target/AMDGPU/AsmParser/AMDGPUAsmParser.cpp
@@ -1584,8 +1584,8 @@ class AMDGPUAsmParser : public MCTargetAsmParser {
return getFeatureBits()[AMDGPU::FeaturePartialNSAEncoding];
}
- bool hasGloballyAddressableScratch() const {
- return getFeatureBits()[AMDGPU::FeatureGloballyAddressableScratch];
+ bool hasGloballyAddressableScratchSupport() const {
+ return getFeatureBits()[AMDGPU::FeatureGloballyAddressableScratchSupport];
}
unsigned getNSAMaxSize(bool HasSampler = false) const {
@@ -7176,7 +7176,7 @@ bool AMDGPUAsmParser::subtargetHasRegister(const MCRegisterInfo &MRI,
return AMDGPU::hasPrivateApertureRegs(getSTI());
case SRC_FLAT_SCRATCH_BASE_LO:
case SRC_FLAT_SCRATCH_BASE_HI:
- return hasGloballyAddressableScratch();
+ return hasGloballyAddressableScratchSupport();
case SRC_POPS_EXITING_WAVE_ID:
return hasPopsExitingWaveID(getSTI());
case TBA:
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
index 9443420c3d424..6dbc1733dc1ef 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.cpp
@@ -54,6 +54,10 @@ static cl::opt<unsigned>
cl::desc("Number of addresses from which to enable MIMG NSA."),
cl::init(2), cl::Hidden);
+static cl::opt<bool> EnableGloballyAddressableScratch(
+ "amdgpu-globally-addressable-scratch",
+ cl::desc("Enable Globally Addressable Scratch"), cl::init(false));
+
GCNSubtarget::~GCNSubtarget() = default;
static AMDGPUSubtarget::Generation computeDefaultGeneration(const Triple &TT) {
@@ -894,6 +898,11 @@ unsigned GCNSubtarget::getNSAThreshold(const MachineFunction &MF) const {
return NSAThreshold;
}
+bool GCNSubtarget::isGloballyAddressableScratchEnabled() const {
+ return HasGloballyAddressableScratchSupport &&
+ EnableGloballyAddressableScratch;
+}
+
GCNUserSGPRUsageInfo::GCNUserSGPRUsageInfo(const Function &F,
const GCNSubtarget &ST)
: ST(ST) {
diff --git a/llvm/lib/Target/AMDGPU/GCNSubtarget.h b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
index 34fd3cb4f6594..8d3823ecddcc9 100644
--- a/llvm/lib/Target/AMDGPU/GCNSubtarget.h
+++ b/llvm/lib/Target/AMDGPU/GCNSubtarget.h
@@ -1072,6 +1072,8 @@ class GCNSubtarget final : public AMDGPUGenSubtargetInfo,
bool useVMulU64Inst() const {
return hasVMulU64Inst() && !hasSlowMaxMinMulI64Insts();
}
+
+ bool isGloballyAddressableScratchEnabled() const;
};
class GCNUserSGPRUsageInfo {
diff --git a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
index 4a3c9258f1770..381a30b68927a 100644
--- a/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
+++ b/llvm/lib/Target/AMDGPU/SIISelLowering.cpp
@@ -2379,7 +2379,7 @@ bool SITargetLowering::isFreeAddrSpaceCast(unsigned SrcAS,
unsigned DestAS) const {
if (SrcAS == AMDGPUAS::FLAT_ADDRESS) {
if (DestAS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
// Flat -> private requires subtracting src_flat_scratch_base_lo.
return false;
}
@@ -9340,7 +9340,7 @@ SDValue SITargetLowering::getBaseSegmentAperture(unsigned AS, const SDLoc &DL,
const unsigned ApertureRegNo =
IsLDS ? AMDGPU::SRC_SHARED_BASE : AMDGPU::SRC_PRIVATE_BASE;
assert((ApertureRegNo != AMDGPU::SRC_PRIVATE_BASE ||
- !Subtarget->hasGloballyAddressableScratch()) &&
+ !Subtarget->hasGloballyAddressableScratchSupport()) &&
"Cannot use src_private_base with globally addressable scratch!");
// Note: this feature (register) is broken. When used as a 32-bit operand,
// it returns a wrong value (all zeroes?). The real value is in the upper 32
@@ -9430,7 +9430,7 @@ SDValue SITargetLowering::lowerADDRSPACECAST(SDValue Op,
SDValue Ptr = DAG.getNode(ISD::TRUNCATE, SL, MVT::i32, Src);
if (DestAS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
// flat -> private with globally addressable scratch: subtract
// src_flat_scratch_base_lo.
SDValue FlatScratchBaseLo(
@@ -9459,7 +9459,7 @@ SDValue SITargetLowering::lowerADDRSPACECAST(SDValue Op,
SrcAS == AMDGPUAS::PRIVATE_ADDRESS || SrcAS == AMDGPUAS::BARRIER) {
SDValue CvtPtr;
if (SrcAS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
// For wave32: Addr = (TID[4:0] << 52) + FLAT_SCRATCH_BASE + privateAddr
// For wave64: Addr = (TID[5:0] << 51) + FLAT_SCRATCH_BASE + privateAddr
SDValue AllOnes = DAG.getSignedTargetConstant(-1, SL, MVT::i32);
@@ -11520,7 +11520,7 @@ SDValue SITargetLowering::LowerINTRINSIC_WO_CHAIN(SDValue Op,
? AMDGPUAS::LOCAL_ADDRESS
: AMDGPUAS::PRIVATE_ADDRESS;
if (AS == AMDGPUAS::PRIVATE_ADDRESS &&
- Subtarget->hasGloballyAddressableScratch()) {
+ Subtarget->hasGloballyAddressableScratchSupport()) {
SDValue FlatScratchBaseHi(
DAG.getMachineNode(
AMDGPU::S_MOV_B32, DL, MVT::i32,
@@ -20774,8 +20774,8 @@ static bool flatInstrMayAccessPrivate(const Instruction *I) {
static TargetLowering::AtomicExpansionKind
getPrivateAtomicExpansionKind(const GCNSubtarget &STI) {
- // For GAS, lower to flat atomic.
- return STI.hasGloballyAddressableScratch()
+ // If GAS is enabled, scratch atomics need to be expanded to FLAT.
+ return STI.isGloballyAddressableScratchEnabled()
? TargetLowering::AtomicExpansionKind::CustomExpand
: TargetLowering::AtomicExpansionKind::NotAtomic;
}
diff --git a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
index f2f2a1b7d2eaa..e1acad0ddadc9 100644
--- a/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/SIInstrInfo.cpp
@@ -11189,7 +11189,7 @@ SIInstrInfo::getGenericValueUniformity(const MachineInstr &MI) const {
unsigned SrcAS = SrcTy.getAddressSpace();
return SrcAS == AMDGPUAS::PRIVATE_ADDRESS &&
DstAS == AMDGPUAS::FLAT_ADDRESS &&
- ST.hasGloballyAddressableScratch()
+ ST.hasGloballyAddressableScratchSupport()
? ValueUniformity::NeverUniform
: ValueUniformity::Default;
};
diff --git a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
index e818e06e3bb04..a441a256d6742 100644
--- a/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
+++ b/llvm/lib/Target/AMDGPU/SIMemoryLegalizer.cpp
@@ -1025,7 +1025,7 @@ bool SICacheControl::enableCPolBits(const MachineBasicBlock::iterator MI,
}
bool SICacheControl::canAffectGlobalAddrSpace(SIAtomicAddrSpace AS) const {
- assert((!ST.hasGloballyAddressableScratch() ||
+ assert((!ST.isGloballyAddressableScratchEnabled() ||
(AS & SIAtomicAddrSpace::GLOBAL) != SIAtomicAddrSpace::NONE ||
(AS & SIAtomicAddrSpace::SCRATCH) == SIAtomicAddrSpace::NONE) &&
"scratch instructions should already be replaced by flat "
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
index 549c91e3859df..fbf16bac4a1b6 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_agent_unordered_load(
; GFX6-LABEL: private_agent_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
index ecfb348b0e894..73722ee9fa59a 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_cluster_unordered_load(
; GFX6-LABEL: private_cluster_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
index a9c2ed7be80f1..97491515a6875 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-lastuse.ll
@@ -1,7 +1,8 @@
; NOTE: Assertions have been autogenerated by utils/update_llc_test_checks.py
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefix=GFX12 %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefix=GFX12 %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
; RUN: llc -mtriple=amdgpu13.10-amd-amdhsa -O0 < %s | FileCheck --check-prefix=GFX13 %s
; RUN: llc -mtriple=amdgpu13.10-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefix=GFX13 %s
@@ -247,3 +248,6 @@ entry:
!0 = !{i32 1}
declare i32 @llvm.amdgcn.workitem.id.x()
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250-GAS: {{.*}}
+; GFX1250-NOGAS: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
index 7c58c39a5db6e..2113a3751d667 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_nontemporal_load_0(
; GFX6-LABEL: private_nontemporal_load_0:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
index 0b15aacbf3727..607d74f8f79fa 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_singlethread_unordered_load(
; GFX6-LABEL: private_singlethread_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
index 4cc3625fe8d54..22351a19b9121 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_system_unordered_load(
; GFX6-LABEL: private_system_unordered_load:
@@ -225,45 +226,57 @@ define amdgpu_kernel void @private_system_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in unordered, align 4
store i32 %val, ptr addrspace(5) %out
@@ -481,45 +494,57 @@ define amdgpu_kernel void @private_system_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in monotonic, align 4
store i32 %val, ptr addrspace(5) %out
@@ -737,47 +762,59 @@ define amdgpu_kernel void @private_system_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in acquire, align 4
store i32 %val, ptr addrspace(5) %out
@@ -995,49 +1032,61 @@ define amdgpu_kernel void @private_system_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in seq_cst, align 4
store i32 %val, ptr addrspace(5) %out
@@ -1239,44 +1288,55 @@ define amdgpu_kernel void @private_system_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out unordered, align 4
ret void
@@ -1477,44 +1537,55 @@ define amdgpu_kernel void @private_system_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out monotonic, align 4
ret void
@@ -1715,49 +1786,60 @@ define amdgpu_kernel void @private_system_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out release, align 4
ret void
@@ -1958,49 +2040,60 @@ define amdgpu_kernel void @private_system_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out seq_cst, align 4
ret void
@@ -2234,45 +2327,58 @@ define amdgpu_kernel void @private_system_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in monotonic
ret void
@@ -2506,48 +2612,61 @@ define amdgpu_kernel void @private_system_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acquire
ret void
@@ -2781,50 +2900,63 @@ define amdgpu_kernel void @private_system_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in release
ret void
@@ -3058,53 +3190,66 @@ define amdgpu_kernel void @private_system_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acq_rel
ret void
@@ -3338,53 +3483,66 @@ define amdgpu_kernel void @private_system_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in seq_cst
ret void
@@ -3639,48 +3797,62 @@ define amdgpu_kernel void @private_system_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acquire
store i32 %val, ptr addrspace(5) %out, align 4
@@ -3936,53 +4108,67 @@ define amdgpu_kernel void @private_system_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in acq_rel
store i32 %val, ptr addrspace(5) %out, align 4
@@ -4238,53 +4424,67 @@ define amdgpu_kernel void @private_system_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in seq_cst
store i32 %val, ptr addrspace(5) %out, align 4
@@ -4596,53 +4796,70 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in monotonic monotonic
@@ -4954,56 +5171,73 @@ define amdgpu_kernel void @private_system_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acquire monotonic
@@ -5315,58 +5549,75 @@ define amdgpu_kernel void @private_system_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in release monotonic
@@ -5678,61 +5929,78 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acq_rel monotonic
@@ -6044,61 +6312,78 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in seq_cst monotonic
@@ -6410,56 +6695,73 @@ define amdgpu_kernel void @private_system_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in monotonic acquire
@@ -6771,56 +7073,73 @@ define amdgpu_kernel void @private_system_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acquire acquire
@@ -7132,61 +7451,78 @@ define amdgpu_kernel void @private_system_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in release acquire
@@ -7498,61 +7834,78 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acq_rel acquire
@@ -7864,61 +8217,78 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in seq_cst acquire
@@ -8230,61 +8600,78 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in seq_cst seq_cst
@@ -8624,54 +9011,72 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in monotonic monotonic
@@ -9013,56 +9418,74 @@ define amdgpu_kernel void @private_system_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acquire monotonic
@@ -9404,61 +9827,79 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acq_rel monotonic
@@ -9800,61 +10241,79 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in seq_cst monotonic
@@ -10196,56 +10655,74 @@ define amdgpu_kernel void @private_system_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in monotonic acquire
@@ -10587,56 +11064,74 @@ define amdgpu_kernel void @private_system_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acquire acquire
@@ -10978,61 +11473,79 @@ define amdgpu_kernel void @private_system_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in release acquire
@@ -11374,61 +11887,79 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acq_rel acquire
@@ -11770,61 +12301,79 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in seq_cst acquire
@@ -12166,61 +12715,79 @@ define amdgpu_kernel void @private_system_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in monotonic seq_cst
@@ -12562,61 +13129,79 @@ define amdgpu_kernel void @private_system_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acquire seq_cst
@@ -12958,61 +13543,79 @@ define amdgpu_kernel void @private_system_relese_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in release seq_cst
@@ -13354,61 +13957,79 @@ define amdgpu_kernel void @private_system_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in acq_rel seq_cst
@@ -13750,61 +14371,79 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in seq_cst seq_cst
@@ -14024,45 +14663,57 @@ define amdgpu_kernel void @private_system_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") unordered, align 4
store i32 %val, ptr addrspace(5) %out
@@ -14280,45 +14931,57 @@ define amdgpu_kernel void @private_system_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") monotonic, align 4
store i32 %val, ptr addrspace(5) %out
@@ -14536,47 +15199,59 @@ define amdgpu_kernel void @private_system_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") acquire, align 4
store i32 %val, ptr addrspace(5) %out
@@ -14794,49 +15469,61 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %in, ptr addrspace(5) %out) {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("one-as") seq_cst, align 4
store i32 %val, ptr addrspace(5) %out
@@ -15038,44 +15725,55 @@ define amdgpu_kernel void @private_system_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") unordered, align 4
ret void
@@ -15276,44 +15974,55 @@ define amdgpu_kernel void @private_system_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") monotonic, align 4
ret void
@@ -15514,49 +16223,60 @@ define amdgpu_kernel void @private_system_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") release, align 4
ret void
@@ -15757,49 +16477,60 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- i32 %in, ptr addrspace(5) %out) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ i32 %in, ptr addrspace(5) %out) {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("one-as") seq_cst, align 4
ret void
@@ -16033,45 +16764,58 @@ define amdgpu_kernel void @private_system_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") monotonic
ret void
@@ -16305,48 +17049,61 @@ define amdgpu_kernel void @private_system_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acquire
ret void
@@ -16580,50 +17337,63 @@ define amdgpu_kernel void @private_system_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") release
ret void
@@ -16857,53 +17627,66 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acq_rel
ret void
@@ -17137,53 +17920,66 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") seq_cst
ret void
@@ -17438,48 +18234,62 @@ define amdgpu_kernel void @private_system_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acquire
store i32 %val, ptr addrspace(5) %out, align 4
@@ -17735,53 +18545,67 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") acq_rel
store i32 %val, ptr addrspace(5) %out, align 4
@@ -18037,53 +18861,67 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in) {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("one-as") seq_cst
store i32 %val, ptr addrspace(5) %out, align 4
@@ -18395,53 +19233,70 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") monotonic monotonic
@@ -18753,56 +19608,73 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acquire monotonic
@@ -19114,58 +19986,75 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") release monotonic
@@ -19477,61 +20366,78 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acq_rel monotonic
@@ -19843,61 +20749,78 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") seq_cst monotonic
@@ -20209,56 +21132,73 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") monotonic acquire
@@ -20570,56 +21510,73 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acquire acquire
@@ -20931,61 +21888,78 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") release acquire
@@ -21297,61 +22271,78 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acq_rel acquire
@@ -21663,61 +22654,78 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") seq_cst acquire
@@ -22029,61 +23037,78 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") monotonic seq_cst
@@ -22395,61 +23420,78 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acquire seq_cst
@@ -22761,61 +23803,78 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") release seq_cst
@@ -23127,61 +24186,78 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acq_rel seq_cst
@@ -23493,61 +24569,78 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") seq_cst seq_cst
@@ -23887,54 +24980,72 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_ret_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") monotonic monotonic
@@ -24276,56 +25387,74 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acquire monotonic
@@ -24667,59 +25796,77 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") release monotonic
@@ -25061,61 +26208,79 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acq_rel monotonic
@@ -25457,61 +26622,79 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") seq_cst monotonic
@@ -25853,56 +27036,74 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") monotonic acquire
@@ -26244,56 +27445,74 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acquire acquire
@@ -26635,61 +27854,79 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") release acquire
@@ -27031,61 +28268,79 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acq_rel acquire
@@ -27427,61 +28682,79 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") seq_cst acquire
@@ -27823,61 +29096,79 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") monotonic seq_cst
@@ -28219,61 +29510,79 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acquire seq_cst
@@ -28615,61 +29924,79 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") release seq_cst
@@ -29011,61 +30338,79 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") acq_rel seq_cst
@@ -29407,61 +30752,79 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SYS
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
- ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
+; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SYS
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
+ ptr addrspace(5) %out, i32 %in, i32 %old) {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
%val = cmpxchg volatile ptr addrspace(5) %gep, i32 %old, i32 %in syncscope("one-as") seq_cst seq_cst
@@ -29469,5 +30832,5 @@ entry:
store i32 %val0, ptr addrspace(5) %out, align 4
ret void
}
-
-attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
index 3d6e9e7563cd1..2793ff3883857 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-volatile.ll
@@ -8,7 +8,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_volatile_load_0(
; GFX6-LABEL: private_volatile_load_0:
@@ -877,3 +878,6 @@ entry:
}
declare i32 @llvm.amdgcn.workitem.id.x()
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250-GAS: {{.*}}
+; GFX1250-NOGAS: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
index c9cdf41b26eff..48dd47290bc1c 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_wavefront_unordered_load(
; GFX6-LABEL: private_wavefront_unordered_load:
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
index a2ac7b88ceddb..c6e1184880df8 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
@@ -12,7 +12,8 @@
; RUN: llc -mtriple=amdgpu11.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX11-CU %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX12-WGP %s
; RUN: llc -mtriple=amdgpu12.00-amd-amdhsa -O0 -mattr=+cumode < %s | FileCheck --check-prefixes=GFX12-CU %s
-; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250 %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 < %s | FileCheck --check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: llc -mtriple=amdgpu12.50-amd-amdhsa -O0 -amdgpu-globally-addressable-scratch < %s | FileCheck --check-prefixes=GFX1250,GFX1250-GAS %s
define amdgpu_kernel void @private_workgroup_unordered_load(
; GFX6-LABEL: private_workgroup_unordered_load:
diff --git a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
index 2836ca080df84..c77e06a641ad1 100644
--- a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
+++ b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
@@ -1,6 +1,7 @@
; NOTE: Assertions have been autogenerated by utils/update_test_checks.py UTC_ARGS: --version 5
; RUN: opt -S -mtriple=amdgpu12.00-amd-amdhsa -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1200 %s
-; RUN: opt -S -mtriple=amdgpu12.50-amd-amdhsa -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250 %s
+; RUN: opt -S -mtriple=amdgpu12.50-amd-amdhsa -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250,GFX1250-NOGAS %s
+; RUN: opt -S -mtriple=amdgpu12.50-amd-amdhsa -amdgpu-globally-addressable-scratch -passes='require<libcall-lowering-info>,atomic-expand' %s | FileCheck -check-prefixes=GFX1250,GFX1250-GAS %s
define void @system_atomic_store_unordered_float(ptr addrspace(5) %addr, float %val) {
; GFX1200-LABEL: define void @system_atomic_store_unordered_float(
@@ -230,6 +231,11 @@ define float @system_atomicrmw_fminimum_acq_rel_float(ptr addrspace(5) %addr, fl
; GFX1200: [[META0]] = !{i32 1}
; GFX1200: [[META1]] = !{}
;.
-; GFX1250: [[META0]] = !{i32 1}
-; GFX1250: [[META1]] = !{}
+; GFX1250-NOGAS: [[META0]] = !{i32 1}
+; GFX1250-NOGAS: [[META1]] = !{}
;.
+; GFX1250-GAS: [[META0]] = !{i32 1}
+; GFX1250-GAS: [[META1]] = !{}
+;.
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
>From 900aa4c9be44df88281dd945f4affa6c85f6df52 Mon Sep 17 00:00:00 2001
From: pvanhout <pierre.vanhoutryve at amd.com>
Date: Wed, 9 Sep 2026 11:59:23 +0200
Subject: [PATCH 2/2] Rebase
---
.../Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp | 2 +-
.../AMDGPU/memory-legalizer-private-agent.ll | 10806 +++++++++-------
.../memory-legalizer-private-cluster.ll | 10364 +++++++++------
.../memory-legalizer-private-nontemporal.ll | 3 +
.../memory-legalizer-private-singlethread.ll | 10010 ++++++++------
.../AMDGPU/memory-legalizer-private-system.ll | 522 +
.../memory-legalizer-private-wavefront.ll | 10010 ++++++++------
.../memory-legalizer-private-workgroup.ll | 10314 +++++++++------
.../AMDGPU/expand-atomic-private-gas.ll | 209 +-
9 files changed, 31402 insertions(+), 20838 deletions(-)
diff --git a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
index a5d96bedf72af..f8a140643cbed 100644
--- a/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
+++ b/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp
@@ -2556,7 +2556,7 @@ bool hasPopsExitingWaveID(const MCSubtargetInfo &STI) {
bool hasPrivateApertureRegs(const MCSubtargetInfo &STI) {
return STI.hasFeature(AMDGPU::FeatureApertureRegs) &&
- !STI.hasFeature(AMDGPU::FeatureGloballyAddressableScratch);
+ !STI.hasFeature(AMDGPU::FeatureGloballyAddressableScratchSupport);
}
bool isGFX10(const MCSubtargetInfo &STI) {
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
index fbf16bac4a1b6..95f481b5335a0 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-agent.ll
@@ -226,44 +226,62 @@ define amdgpu_kernel void @private_agent_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") unordered, align 4
@@ -482,44 +500,62 @@ define amdgpu_kernel void @private_agent_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") monotonic, align 4
@@ -738,46 +774,64 @@ define amdgpu_kernel void @private_agent_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") acquire, align 4
@@ -996,48 +1050,66 @@ define amdgpu_kernel void @private_agent_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent") seq_cst, align 4
@@ -1240,43 +1312,60 @@ define amdgpu_kernel void @private_agent_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") unordered, align 4
@@ -1478,43 +1567,60 @@ define amdgpu_kernel void @private_agent_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") monotonic, align 4
@@ -1716,48 +1822,65 @@ define amdgpu_kernel void @private_agent_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") release, align 4
@@ -1959,48 +2082,65 @@ define amdgpu_kernel void @private_agent_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent") seq_cst, align 4
@@ -2235,44 +2375,63 @@ define amdgpu_kernel void @private_agent_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") monotonic
@@ -2507,47 +2666,66 @@ define amdgpu_kernel void @private_agent_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acquire
@@ -2782,49 +2960,68 @@ define amdgpu_kernel void @private_agent_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") release
@@ -3059,52 +3256,71 @@ define amdgpu_kernel void @private_agent_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acq_rel
@@ -3339,52 +3555,71 @@ define amdgpu_kernel void @private_agent_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") seq_cst
@@ -3640,47 +3875,67 @@ define amdgpu_kernel void @private_agent_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acquire
@@ -3937,52 +4192,72 @@ define amdgpu_kernel void @private_agent_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") acq_rel
@@ -4239,52 +4514,72 @@ define amdgpu_kernel void @private_agent_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent") seq_cst
@@ -4597,52 +4892,75 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4955,55 +5273,78 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5316,57 +5657,80 @@ define amdgpu_kernel void @private_agent_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5679,60 +6043,83 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6045,60 +6432,83 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6411,55 +6821,78 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6772,55 +7205,78 @@ define amdgpu_kernel void @private_agent_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7133,60 +7589,83 @@ define amdgpu_kernel void @private_agent_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7499,60 +7978,83 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7865,60 +8367,83 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8231,60 +8756,83 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8597,60 +9145,83 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8963,60 +9534,83 @@ define amdgpu_kernel void @private_agent_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9329,60 +9923,83 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9695,60 +10312,83 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10089,53 +10729,77 @@ define amdgpu_kernel void @private_agent_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10478,55 +11142,79 @@ define amdgpu_kernel void @private_agent_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10869,58 +11557,82 @@ define amdgpu_kernel void @private_agent_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11263,60 +11975,84 @@ define amdgpu_kernel void @private_agent_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11659,60 +12395,84 @@ define amdgpu_kernel void @private_agent_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12055,55 +12815,79 @@ define amdgpu_kernel void @private_agent_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12446,55 +13230,79 @@ define amdgpu_kernel void @private_agent_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12837,60 +13645,84 @@ define amdgpu_kernel void @private_agent_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13233,60 +14065,84 @@ define amdgpu_kernel void @private_agent_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13629,60 +14485,84 @@ define amdgpu_kernel void @private_agent_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14025,60 +14905,84 @@ define amdgpu_kernel void @private_agent_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14421,60 +15325,84 @@ define amdgpu_kernel void @private_agent_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14817,60 +15745,84 @@ define amdgpu_kernel void @private_agent_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15213,60 +16165,84 @@ define amdgpu_kernel void @private_agent_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15609,60 +16585,84 @@ define amdgpu_kernel void @private_agent_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15883,44 +16883,62 @@ define amdgpu_kernel void @private_agent_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") unordered, align 4
@@ -16139,44 +17157,62 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") monotonic, align 4
@@ -16395,46 +17431,64 @@ define amdgpu_kernel void @private_agent_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") acquire, align 4
@@ -16653,48 +17707,66 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("agent-one-as") seq_cst, align 4
@@ -16897,43 +17969,60 @@ define amdgpu_kernel void @private_agent_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") unordered, align 4
@@ -17135,43 +18224,60 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") monotonic, align 4
@@ -17373,48 +18479,65 @@ define amdgpu_kernel void @private_agent_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") release, align 4
@@ -17616,48 +18739,65 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("agent-one-as") seq_cst, align 4
@@ -17892,44 +19032,63 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") monotonic
@@ -18164,47 +19323,66 @@ define amdgpu_kernel void @private_agent_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acquire
@@ -18439,49 +19617,68 @@ define amdgpu_kernel void @private_agent_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") release
@@ -18716,52 +19913,71 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acq_rel
@@ -18996,52 +20212,71 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") seq_cst
@@ -19297,47 +20532,67 @@ define amdgpu_kernel void @private_agent_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acquire
@@ -19594,52 +20849,72 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") acq_rel
@@ -19896,52 +21171,72 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("agent-one-as") seq_cst
@@ -20254,52 +21549,75 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20612,55 +21930,78 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20973,57 +22314,80 @@ define amdgpu_kernel void @private_agent_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21336,60 +22700,83 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21702,60 +23089,83 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22068,55 +23478,78 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22429,55 +23862,78 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22790,60 +24246,83 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23156,60 +24635,83 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23522,60 +25024,83 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23888,60 +25413,83 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24254,60 +25802,83 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24620,60 +26191,83 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24986,60 +26580,83 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25352,60 +26969,83 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25746,53 +27386,77 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26135,55 +27799,79 @@ define amdgpu_kernel void @private_agent_one_as_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26526,60 +28214,84 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26922,60 +28634,84 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27318,55 +29054,79 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27709,55 +29469,79 @@ define amdgpu_kernel void @private_agent_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28100,60 +29884,84 @@ define amdgpu_kernel void @private_agent_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28496,60 +30304,84 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28892,60 +30724,84 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29288,60 +31144,84 @@ define amdgpu_kernel void @private_agent_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29684,60 +31564,84 @@ define amdgpu_kernel void @private_agent_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30080,60 +31984,84 @@ define amdgpu_kernel void @private_agent_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30476,60 +32404,84 @@ define amdgpu_kernel void @private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30872,60 +32824,84 @@ define amdgpu_kernel void @private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_wb scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_DEV
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_agent_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_wb scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_DEV
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30936,3 +32912,5 @@ entry:
}
attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
index 73722ee9fa59a..1edc5d70bb516 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-cluster.ll
@@ -226,44 +226,62 @@ define amdgpu_kernel void @private_cluster_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") unordered, align 4
@@ -482,44 +500,62 @@ define amdgpu_kernel void @private_cluster_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") monotonic, align 4
@@ -738,45 +774,63 @@ define amdgpu_kernel void @private_cluster_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") acquire, align 4
@@ -995,47 +1049,65 @@ define amdgpu_kernel void @private_cluster_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster") seq_cst, align 4
@@ -1238,43 +1310,60 @@ define amdgpu_kernel void @private_cluster_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") unordered, align 4
@@ -1476,43 +1565,60 @@ define amdgpu_kernel void @private_cluster_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") monotonic, align 4
@@ -1714,45 +1820,62 @@ define amdgpu_kernel void @private_cluster_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") release, align 4
@@ -1954,45 +2077,62 @@ define amdgpu_kernel void @private_cluster_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster") seq_cst, align 4
@@ -2227,44 +2367,63 @@ define amdgpu_kernel void @private_cluster_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") monotonic
@@ -2499,46 +2658,65 @@ define amdgpu_kernel void @private_cluster_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acquire
@@ -2773,46 +2951,65 @@ define amdgpu_kernel void @private_cluster_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") release
@@ -3047,48 +3244,67 @@ define amdgpu_kernel void @private_cluster_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acq_rel
@@ -3323,48 +3539,67 @@ define amdgpu_kernel void @private_cluster_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") seq_cst
@@ -3620,46 +3855,66 @@ define amdgpu_kernel void @private_cluster_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acquire
@@ -3916,48 +4171,68 @@ define amdgpu_kernel void @private_cluster_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") acq_rel
@@ -4214,48 +4489,68 @@ define amdgpu_kernel void @private_cluster_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster") seq_cst
@@ -4568,52 +4863,75 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4926,54 +5244,77 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5286,54 +5627,77 @@ define amdgpu_kernel void @private_cluster_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5646,56 +6010,79 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6008,56 +6395,79 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6370,54 +6780,77 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6730,54 +7163,77 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7090,56 +7546,79 @@ define amdgpu_kernel void @private_cluster_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7452,56 +7931,79 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7814,56 +8316,79 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8176,56 +8701,79 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8538,56 +9086,79 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8900,56 +9471,79 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9262,56 +9856,79 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9624,56 +10241,79 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10014,53 +10654,77 @@ define amdgpu_kernel void @private_cluster_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10403,54 +11067,78 @@ define amdgpu_kernel void @private_cluster_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10793,55 +11481,79 @@ define amdgpu_kernel void @private_cluster_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11184,56 +11896,80 @@ define amdgpu_kernel void @private_cluster_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11576,56 +12312,80 @@ define amdgpu_kernel void @private_cluster_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11968,54 +12728,78 @@ define amdgpu_kernel void @private_cluster_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12358,54 +13142,78 @@ define amdgpu_kernel void @private_cluster_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12748,56 +13556,80 @@ define amdgpu_kernel void @private_cluster_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13140,56 +13972,80 @@ define amdgpu_kernel void @private_cluster_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13532,56 +14388,80 @@ define amdgpu_kernel void @private_cluster_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13924,56 +14804,80 @@ define amdgpu_kernel void @private_cluster_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14316,56 +15220,80 @@ define amdgpu_kernel void @private_cluster_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14708,56 +15636,80 @@ define amdgpu_kernel void @private_cluster_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15100,56 +16052,80 @@ define amdgpu_kernel void @private_cluster_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15492,56 +16468,80 @@ define amdgpu_kernel void @private_cluster_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15762,44 +16762,62 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") unordered, align 4
@@ -16018,44 +17036,62 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") monotonic, align 4
@@ -16274,46 +17310,64 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") acquire, align 4
@@ -16532,48 +17586,66 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("cluster-one-as") seq_cst, align 4
@@ -16776,43 +17848,60 @@ define amdgpu_kernel void @private_cluster_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") unordered, align 4
@@ -17014,43 +18103,60 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") monotonic, align 4
@@ -17252,45 +18358,62 @@ define amdgpu_kernel void @private_cluster_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") release, align 4
@@ -17492,45 +18615,62 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("cluster-one-as") seq_cst, align 4
@@ -17765,44 +18905,63 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") monotonic
@@ -18037,46 +19196,65 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acquire
@@ -18311,46 +19489,65 @@ define amdgpu_kernel void @private_cluster_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") release
@@ -18585,48 +19782,67 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acq_rel
@@ -18861,48 +20077,67 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2 scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") seq_cst
@@ -19158,47 +20393,67 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acquire
@@ -19455,49 +20710,69 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") acq_rel
@@ -19754,49 +21029,69 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("cluster-one-as") seq_cst
@@ -20109,52 +21404,75 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20467,54 +21785,77 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20827,54 +22168,77 @@ define amdgpu_kernel void @private_cluster_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21187,56 +22551,79 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21549,56 +22936,79 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21911,54 +23321,77 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22271,54 +23704,77 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22631,56 +24087,79 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22993,56 +24472,79 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23355,56 +24857,79 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23717,56 +25242,79 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24079,56 +25627,79 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24441,56 +26012,79 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24803,56 +26397,79 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25165,56 +26782,79 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25555,53 +27195,77 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25944,55 +27608,79 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26335,57 +28023,81 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26728,57 +28440,81 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27121,55 +28857,79 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27512,55 +29272,79 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27903,57 +29687,81 @@ define amdgpu_kernel void @private_cluster_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28296,57 +30104,81 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28689,57 +30521,81 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29082,57 +30938,81 @@ define amdgpu_kernel void @private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29475,57 +31355,81 @@ define amdgpu_kernel void @private_cluster_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29868,57 +31772,81 @@ define amdgpu_kernel void @private_cluster_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30261,57 +32189,81 @@ define amdgpu_kernel void @private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30654,57 +32606,81 @@ define amdgpu_kernel void @private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: global_inv scope:SCOPE_SE
-; GFX1250-NEXT: s_wait_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_cluster_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: global_inv scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_wait_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30715,3 +32691,5 @@ entry:
}
attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
index 2113a3751d667..b29e3027d9c9e 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-nontemporal.ll
@@ -1371,3 +1371,6 @@ entry:
declare i32 @llvm.amdgcn.workitem.id.x()
attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250-GAS: {{.*}}
+; GFX1250-NOGAS: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
index 607d74f8f79fa..e89d5eefd8c98 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-singlethread.ll
@@ -226,44 +226,62 @@ define amdgpu_kernel void @private_singlethread_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") unordered, align 4
@@ -482,44 +500,62 @@ define amdgpu_kernel void @private_singlethread_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") monotonic, align 4
@@ -738,44 +774,62 @@ define amdgpu_kernel void @private_singlethread_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") acquire, align 4
@@ -994,44 +1048,62 @@ define amdgpu_kernel void @private_singlethread_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread") seq_cst, align 4
@@ -1234,43 +1306,60 @@ define amdgpu_kernel void @private_singlethread_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") unordered, align 4
@@ -1472,43 +1561,60 @@ define amdgpu_kernel void @private_singlethread_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") monotonic, align 4
@@ -1710,43 +1816,60 @@ define amdgpu_kernel void @private_singlethread_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") release, align 4
@@ -1948,43 +2071,60 @@ define amdgpu_kernel void @private_singlethread_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread") seq_cst, align 4
@@ -2219,44 +2359,63 @@ define amdgpu_kernel void @private_singlethread_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") monotonic
@@ -2491,44 +2650,63 @@ define amdgpu_kernel void @private_singlethread_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acquire
@@ -2763,44 +2941,63 @@ define amdgpu_kernel void @private_singlethread_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") release
@@ -3035,44 +3232,63 @@ define amdgpu_kernel void @private_singlethread_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acq_rel
@@ -3307,44 +3523,63 @@ define amdgpu_kernel void @private_singlethread_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") seq_cst
@@ -3600,45 +3835,65 @@ define amdgpu_kernel void @private_singlethread_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acquire
@@ -3895,45 +4150,65 @@ define amdgpu_kernel void @private_singlethread_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") acq_rel
@@ -4190,45 +4465,65 @@ define amdgpu_kernel void @private_singlethread_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread") seq_cst
@@ -4541,52 +4836,75 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4899,52 +5217,75 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5257,52 +5598,75 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5615,52 +5979,75 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5973,52 +6360,75 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6331,52 +6741,75 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6689,52 +7122,75 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7047,52 +7503,75 @@ define amdgpu_kernel void @private_singlethread_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7405,52 +7884,75 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7763,52 +8265,75 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8121,52 +8646,75 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8479,52 +9027,75 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8837,52 +9408,75 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9195,52 +9789,75 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9553,52 +10170,75 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9939,53 +10579,77 @@ define amdgpu_kernel void @private_singlethread_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10328,53 +10992,77 @@ define amdgpu_kernel void @private_singlethread_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10717,53 +11405,77 @@ define amdgpu_kernel void @private_singlethread_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11106,53 +11818,77 @@ define amdgpu_kernel void @private_singlethread_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11495,53 +12231,77 @@ define amdgpu_kernel void @private_singlethread_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11884,53 +12644,77 @@ define amdgpu_kernel void @private_singlethread_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12273,53 +13057,77 @@ define amdgpu_kernel void @private_singlethread_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12662,53 +13470,77 @@ define amdgpu_kernel void @private_singlethread_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13051,53 +13883,77 @@ define amdgpu_kernel void @private_singlethread_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13440,53 +14296,77 @@ define amdgpu_kernel void @private_singlethread_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13829,53 +14709,77 @@ define amdgpu_kernel void @private_singlethread_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14218,53 +15122,77 @@ define amdgpu_kernel void @private_singlethread_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14607,53 +15535,77 @@ define amdgpu_kernel void @private_singlethread_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14996,53 +15948,77 @@ define amdgpu_kernel void @private_singlethread_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15385,53 +16361,77 @@ define amdgpu_kernel void @private_singlethread_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15652,44 +16652,62 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") unordered, align 4
@@ -15908,44 +16926,62 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") monotonic, align 4
@@ -16164,44 +17200,62 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") acquire, align 4
@@ -16420,44 +17474,62 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("singlethread-one-as") seq_cst, align 4
@@ -16660,43 +17732,60 @@ define amdgpu_kernel void @private_singlethread_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") unordered, align 4
@@ -16898,43 +17987,60 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") monotonic, align 4
@@ -17136,43 +18242,60 @@ define amdgpu_kernel void @private_singlethread_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") release, align 4
@@ -17374,43 +18497,60 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("singlethread-one-as") seq_cst, align 4
@@ -17645,44 +18785,63 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") monotonic
@@ -17917,44 +19076,63 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acquire
@@ -18189,44 +19367,63 @@ define amdgpu_kernel void @private_singlethread_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") release
@@ -18461,44 +19658,63 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acq_rel
@@ -18733,44 +19949,63 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") seq_cst
@@ -19026,45 +20261,65 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acquire
@@ -19321,45 +20576,65 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") acq_rel
@@ -19616,45 +20891,65 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("singlethread-one-as") seq_cst
@@ -19967,52 +21262,75 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20325,52 +21643,75 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20683,52 +22024,75 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21041,52 +22405,75 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21399,52 +22786,75 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21757,52 +23167,75 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22115,52 +23548,75 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22473,52 +23929,75 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22831,52 +24310,75 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23189,52 +24691,75 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23547,52 +25072,75 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_cmpxchg
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23905,52 +25453,75 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24263,52 +25834,75 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24621,52 +26215,75 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24979,52 +26596,75 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25365,53 +27005,77 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_monotonic_ret_c
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25754,53 +27418,77 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26143,53 +27831,77 @@ define amdgpu_kernel void @private_singlethread_one_as_release_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26532,53 +28244,77 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26921,53 +28657,77 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_monotonic_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27310,53 +29070,77 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_acquire_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27699,53 +29483,77 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28088,53 +29896,77 @@ define amdgpu_kernel void @private_singlethread_one_as_release_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28477,53 +30309,77 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28866,53 +30722,77 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_acquire_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29255,53 +31135,77 @@ define amdgpu_kernel void @private_singlethread_one_as_monotonic_seq_cst_ret_cmp
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29644,53 +31548,77 @@ define amdgpu_kernel void @private_singlethread_one_as_acquire_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30033,53 +31961,77 @@ define amdgpu_kernel void @private_singlethread_one_as_release_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30422,53 +32374,77 @@ define amdgpu_kernel void @private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30811,53 +32787,77 @@ define amdgpu_kernel void @private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxc
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_singlethread_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30868,3 +32868,5 @@ entry:
}
attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
index 22351a19b9121..969415aa4df59 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-system.ll
@@ -229,6 +229,9 @@ define amdgpu_kernel void @private_system_unordered_load(
; GFX1250-NOGAS-LABEL: private_system_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -244,6 +247,9 @@ define amdgpu_kernel void @private_system_unordered_load(
; GFX1250-GAS-LABEL: private_system_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -497,6 +503,9 @@ define amdgpu_kernel void @private_system_monotonic_load(
; GFX1250-NOGAS-LABEL: private_system_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -512,6 +521,9 @@ define amdgpu_kernel void @private_system_monotonic_load(
; GFX1250-GAS-LABEL: private_system_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -765,6 +777,9 @@ define amdgpu_kernel void @private_system_acquire_load(
; GFX1250-NOGAS-LABEL: private_system_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -780,6 +795,9 @@ define amdgpu_kernel void @private_system_acquire_load(
; GFX1250-GAS-LABEL: private_system_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1035,6 +1053,9 @@ define amdgpu_kernel void @private_system_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1050,6 +1071,9 @@ define amdgpu_kernel void @private_system_seq_cst_load(
; GFX1250-GAS-LABEL: private_system_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1291,6 +1315,9 @@ define amdgpu_kernel void @private_system_unordered_store(
; GFX1250-NOGAS-LABEL: private_system_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1305,6 +1332,9 @@ define amdgpu_kernel void @private_system_unordered_store(
; GFX1250-GAS-LABEL: private_system_unordered_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1540,6 +1570,9 @@ define amdgpu_kernel void @private_system_monotonic_store(
; GFX1250-NOGAS-LABEL: private_system_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1554,6 +1587,9 @@ define amdgpu_kernel void @private_system_monotonic_store(
; GFX1250-GAS-LABEL: private_system_monotonic_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1789,6 +1825,9 @@ define amdgpu_kernel void @private_system_release_store(
; GFX1250-NOGAS-LABEL: private_system_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -1803,6 +1842,9 @@ define amdgpu_kernel void @private_system_release_store(
; GFX1250-GAS-LABEL: private_system_release_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2043,6 +2085,9 @@ define amdgpu_kernel void @private_system_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2057,6 +2102,9 @@ define amdgpu_kernel void @private_system_seq_cst_store(
; GFX1250-GAS-LABEL: private_system_seq_cst_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2330,6 +2378,9 @@ define amdgpu_kernel void @private_system_monotonic_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_monotonic_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2346,6 +2397,9 @@ define amdgpu_kernel void @private_system_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_system_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2615,6 +2669,9 @@ define amdgpu_kernel void @private_system_acquire_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_acquire_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2631,6 +2688,9 @@ define amdgpu_kernel void @private_system_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_system_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2903,6 +2963,9 @@ define amdgpu_kernel void @private_system_release_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_release_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -2919,6 +2982,9 @@ define amdgpu_kernel void @private_system_release_atomicrmw(
; GFX1250-GAS-LABEL: private_system_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -3193,6 +3259,9 @@ define amdgpu_kernel void @private_system_acq_rel_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_acq_rel_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -3209,6 +3278,9 @@ define amdgpu_kernel void @private_system_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_system_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -3486,6 +3558,9 @@ define amdgpu_kernel void @private_system_seq_cst_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -3502,6 +3577,9 @@ define amdgpu_kernel void @private_system_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_system_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -3800,6 +3878,9 @@ define amdgpu_kernel void @private_system_acquire_ret_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_acquire_ret_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -3817,6 +3898,9 @@ define amdgpu_kernel void @private_system_acquire_ret_atomicrmw(
; GFX1250-GAS-LABEL: private_system_acquire_ret_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -4111,6 +4195,9 @@ define amdgpu_kernel void @private_system_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_acq_rel_ret_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -4128,6 +4215,9 @@ define amdgpu_kernel void @private_system_acq_rel_ret_atomicrmw(
; GFX1250-GAS-LABEL: private_system_acq_rel_ret_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -4427,6 +4517,9 @@ define amdgpu_kernel void @private_system_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_ret_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -4444,6 +4537,9 @@ define amdgpu_kernel void @private_system_seq_cst_ret_atomicrmw(
; GFX1250-GAS-LABEL: private_system_seq_cst_ret_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -4799,6 +4895,9 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_monotonic_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -4819,6 +4918,9 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -5174,6 +5276,9 @@ define amdgpu_kernel void @private_system_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acquire_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -5194,6 +5299,9 @@ define amdgpu_kernel void @private_system_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -5552,6 +5660,9 @@ define amdgpu_kernel void @private_system_release_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_release_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -5572,6 +5683,9 @@ define amdgpu_kernel void @private_system_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -5932,6 +6046,9 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acq_rel_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -5952,6 +6069,9 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -6315,6 +6435,9 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -6335,6 +6458,9 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -6698,6 +6824,9 @@ define amdgpu_kernel void @private_system_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_monotonic_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -6718,6 +6847,9 @@ define amdgpu_kernel void @private_system_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -7076,6 +7208,9 @@ define amdgpu_kernel void @private_system_acquire_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acquire_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -7096,6 +7231,9 @@ define amdgpu_kernel void @private_system_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -7454,6 +7592,9 @@ define amdgpu_kernel void @private_system_release_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_release_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -7474,6 +7615,9 @@ define amdgpu_kernel void @private_system_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -7837,6 +7981,9 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acq_rel_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -7857,6 +8004,9 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -8220,6 +8370,9 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -8240,6 +8393,9 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -8603,6 +8759,9 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -8623,6 +8782,9 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -9014,6 +9176,9 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -9035,6 +9200,9 @@ define amdgpu_kernel void @private_system_monotonic_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_monotonic_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -9421,6 +9589,9 @@ define amdgpu_kernel void @private_system_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -9442,6 +9613,9 @@ define amdgpu_kernel void @private_system_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acquire_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -9830,6 +10004,9 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -9851,6 +10028,9 @@ define amdgpu_kernel void @private_system_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acq_rel_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -10244,6 +10424,9 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -10265,6 +10448,9 @@ define amdgpu_kernel void @private_system_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -10658,6 +10844,9 @@ define amdgpu_kernel void @private_system_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -10679,6 +10868,9 @@ define amdgpu_kernel void @private_system_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_monotonic_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -11067,6 +11259,9 @@ define amdgpu_kernel void @private_system_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acquire_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -11088,6 +11283,9 @@ define amdgpu_kernel void @private_system_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acquire_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -11476,6 +11674,9 @@ define amdgpu_kernel void @private_system_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_release_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -11497,6 +11698,9 @@ define amdgpu_kernel void @private_system_release_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_release_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -11890,6 +12094,9 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -11911,6 +12118,9 @@ define amdgpu_kernel void @private_system_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acq_rel_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -12304,6 +12514,9 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -12325,6 +12538,9 @@ define amdgpu_kernel void @private_system_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -12718,6 +12934,9 @@ define amdgpu_kernel void @private_system_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -12739,6 +12958,9 @@ define amdgpu_kernel void @private_system_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_monotonic_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -13132,6 +13354,9 @@ define amdgpu_kernel void @private_system_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -13153,6 +13378,9 @@ define amdgpu_kernel void @private_system_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acquire_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -13546,6 +13774,9 @@ define amdgpu_kernel void @private_system_relese_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -13567,6 +13798,9 @@ define amdgpu_kernel void @private_system_relese_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_relese_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -13960,6 +14194,9 @@ define amdgpu_kernel void @private_system_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -13981,6 +14218,9 @@ define amdgpu_kernel void @private_system_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_acq_rel_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -14374,6 +14614,9 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -14395,6 +14638,9 @@ define amdgpu_kernel void @private_system_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_seq_cst_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -14666,6 +14912,9 @@ define amdgpu_kernel void @private_system_one_as_unordered_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -14681,6 +14930,9 @@ define amdgpu_kernel void @private_system_one_as_unordered_load(
; GFX1250-GAS-LABEL: private_system_one_as_unordered_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -14934,6 +15186,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -14949,6 +15204,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_load(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15202,6 +15460,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15217,6 +15478,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_load(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15472,6 +15736,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_load(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_load:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15487,6 +15754,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_load(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_load:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15728,6 +15998,9 @@ define amdgpu_kernel void @private_system_one_as_unordered_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_unordered_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15742,6 +16015,9 @@ define amdgpu_kernel void @private_system_one_as_unordered_store(
; GFX1250-GAS-LABEL: private_system_one_as_unordered_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15977,6 +16253,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -15991,6 +16270,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_store(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -16226,6 +16508,9 @@ define amdgpu_kernel void @private_system_one_as_release_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -16240,6 +16525,9 @@ define amdgpu_kernel void @private_system_one_as_release_store(
; GFX1250-GAS-LABEL: private_system_one_as_release_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -16480,6 +16768,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_store(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_store:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -16494,6 +16785,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_store(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_store:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -16767,6 +17061,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -16783,6 +17080,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17052,6 +17352,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17068,6 +17371,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17340,6 +17646,9 @@ define amdgpu_kernel void @private_system_one_as_release_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17356,6 +17665,9 @@ define amdgpu_kernel void @private_system_one_as_release_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_release_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17630,6 +17942,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17646,6 +17961,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17923,6 +18241,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -17939,6 +18260,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -18237,6 +18561,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_ret_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_ret_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -18254,6 +18581,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_ret_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_ret_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -18548,6 +18878,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_ret_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -18565,6 +18898,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_ret_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_ret_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -18864,6 +19200,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_ret_atomicrmw(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -18881,6 +19220,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_ret_atomicrmw(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_ret_atomicrmw:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -19236,6 +19578,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -19256,6 +19601,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -19611,6 +19959,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -19631,6 +19982,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -19989,6 +20343,9 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -20009,6 +20366,9 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -20369,6 +20729,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -20389,6 +20752,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -20752,6 +21118,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -20772,6 +21141,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_monotonic_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -21135,6 +21507,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -21155,6 +21530,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -21513,6 +21891,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -21533,6 +21914,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -21891,6 +22275,9 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -21911,6 +22298,9 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -22274,6 +22664,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -22294,6 +22687,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -22657,6 +23053,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -22677,6 +23076,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_acquire_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -23040,6 +23442,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -23060,6 +23465,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -23423,6 +23831,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -23443,6 +23854,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -23806,6 +24220,9 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -23826,6 +24243,9 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -24189,6 +24609,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -24209,6 +24632,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -24572,6 +24998,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -24592,6 +25021,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_seq_cst_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -24983,6 +25415,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_ret_cmpxchg
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -25004,6 +25439,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_monotonic_ret_cmpxchg
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -25390,6 +25828,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -25411,6 +25852,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -25799,6 +26243,9 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -25820,6 +26267,9 @@ define amdgpu_kernel void @private_system_one_as_release_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -26211,6 +26661,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -26232,6 +26685,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -26625,6 +27081,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -26646,6 +27105,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_monotonic_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_monotonic_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -27039,6 +27501,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -27060,6 +27525,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -27448,6 +27916,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -27469,6 +27940,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -27857,6 +28331,9 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -27878,6 +28355,9 @@ define amdgpu_kernel void @private_system_one_as_release_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -28271,6 +28751,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -28292,6 +28775,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -28685,6 +29171,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -28706,6 +29195,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_acquire_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -29099,6 +29591,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -29120,6 +29615,9 @@ define amdgpu_kernel void @private_system_one_as_monotonic_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_monotonic_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -29513,6 +30011,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -29534,6 +30035,9 @@ define amdgpu_kernel void @private_system_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acquire_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -29927,6 +30431,9 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -29948,6 +30455,9 @@ define amdgpu_kernel void @private_system_one_as_release_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_release_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -30341,6 +30851,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -30362,6 +30875,9 @@ define amdgpu_kernel void @private_system_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_acq_rel_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -30755,6 +31271,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-NOGAS-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
; GFX1250-NOGAS: ; %bb.0: ; %entry
; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
@@ -30776,6 +31295,9 @@ define amdgpu_kernel void @private_system_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX1250-GAS-LABEL: private_system_one_as_seq_cst_seq_cst_ret_cmpxchg:
; GFX1250-GAS: ; %bb.0: ; %entry
; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
index 48dd47290bc1c..8f453c85477fa 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-wavefront.ll
@@ -226,44 +226,62 @@ define amdgpu_kernel void @private_wavefront_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") unordered, align 4
@@ -482,44 +500,62 @@ define amdgpu_kernel void @private_wavefront_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") monotonic, align 4
@@ -738,44 +774,62 @@ define amdgpu_kernel void @private_wavefront_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") acquire, align 4
@@ -994,44 +1048,62 @@ define amdgpu_kernel void @private_wavefront_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront") seq_cst, align 4
@@ -1234,43 +1306,60 @@ define amdgpu_kernel void @private_wavefront_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") unordered, align 4
@@ -1472,43 +1561,60 @@ define amdgpu_kernel void @private_wavefront_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") monotonic, align 4
@@ -1710,43 +1816,60 @@ define amdgpu_kernel void @private_wavefront_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") release, align 4
@@ -1948,43 +2071,60 @@ define amdgpu_kernel void @private_wavefront_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront") seq_cst, align 4
@@ -2219,44 +2359,63 @@ define amdgpu_kernel void @private_wavefront_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") monotonic
@@ -2491,44 +2650,63 @@ define amdgpu_kernel void @private_wavefront_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acquire
@@ -2763,44 +2941,63 @@ define amdgpu_kernel void @private_wavefront_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") release
@@ -3035,44 +3232,63 @@ define amdgpu_kernel void @private_wavefront_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acq_rel
@@ -3307,44 +3523,63 @@ define amdgpu_kernel void @private_wavefront_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") seq_cst
@@ -3600,45 +3835,65 @@ define amdgpu_kernel void @private_wavefront_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acquire
@@ -3895,45 +4150,65 @@ define amdgpu_kernel void @private_wavefront_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") acq_rel
@@ -4190,45 +4465,65 @@ define amdgpu_kernel void @private_wavefront_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront") seq_cst
@@ -4541,52 +4836,75 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4899,52 +5217,75 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5257,52 +5598,75 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5615,52 +5979,75 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5973,52 +6360,75 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6331,52 +6741,75 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6689,52 +7122,75 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7047,52 +7503,75 @@ define amdgpu_kernel void @private_wavefront_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7405,52 +7884,75 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7763,52 +8265,75 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8121,52 +8646,75 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8479,52 +9027,75 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8837,52 +9408,75 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9195,52 +9789,75 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9553,52 +10170,75 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9939,53 +10579,77 @@ define amdgpu_kernel void @private_wavefront_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10328,53 +10992,77 @@ define amdgpu_kernel void @private_wavefront_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10717,53 +11405,77 @@ define amdgpu_kernel void @private_wavefront_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11106,53 +11818,77 @@ define amdgpu_kernel void @private_wavefront_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11495,53 +12231,77 @@ define amdgpu_kernel void @private_wavefront_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11884,53 +12644,77 @@ define amdgpu_kernel void @private_wavefront_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12273,53 +13057,77 @@ define amdgpu_kernel void @private_wavefront_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12662,53 +13470,77 @@ define amdgpu_kernel void @private_wavefront_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13051,53 +13883,77 @@ define amdgpu_kernel void @private_wavefront_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13440,53 +14296,77 @@ define amdgpu_kernel void @private_wavefront_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13829,53 +14709,77 @@ define amdgpu_kernel void @private_wavefront_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14218,53 +15122,77 @@ define amdgpu_kernel void @private_wavefront_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14607,53 +15535,77 @@ define amdgpu_kernel void @private_wavefront_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14996,53 +15948,77 @@ define amdgpu_kernel void @private_wavefront_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15385,53 +16361,77 @@ define amdgpu_kernel void @private_wavefront_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15652,44 +16652,62 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") unordered, align 4
@@ -15908,44 +16926,62 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") monotonic, align 4
@@ -16164,44 +17200,62 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") acquire, align 4
@@ -16420,44 +17474,62 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("wavefront-one-as") seq_cst, align 4
@@ -16660,43 +17732,60 @@ define amdgpu_kernel void @private_wavefront_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") unordered, align 4
@@ -16898,43 +17987,60 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") monotonic, align 4
@@ -17136,43 +18242,60 @@ define amdgpu_kernel void @private_wavefront_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") release, align 4
@@ -17374,43 +18497,60 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("wavefront-one-as") seq_cst, align 4
@@ -17645,44 +18785,63 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") monotonic
@@ -17917,44 +19076,63 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acquire
@@ -18189,44 +19367,63 @@ define amdgpu_kernel void @private_wavefront_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") release
@@ -18461,44 +19658,63 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acq_rel
@@ -18733,44 +19949,63 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") seq_cst
@@ -19026,45 +20261,65 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acquire
@@ -19321,45 +20576,65 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") acq_rel
@@ -19616,45 +20891,65 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("wavefront-one-as") seq_cst
@@ -19967,52 +21262,75 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20325,52 +21643,75 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20683,52 +22024,75 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21041,52 +22405,75 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21399,52 +22786,75 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21757,52 +23167,75 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22115,52 +23548,75 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22473,52 +23929,75 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22831,52 +24310,75 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23189,52 +24691,75 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23547,52 +25072,75 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23905,52 +25453,75 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24263,52 +25834,75 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24621,52 +26215,75 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24979,52 +26596,75 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25365,53 +27005,77 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_monotonic_ret_cmpx
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25754,53 +27418,77 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26143,53 +27831,77 @@ define amdgpu_kernel void @private_wavefront_one_as_release_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26532,53 +28244,77 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26921,53 +28657,77 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27310,53 +29070,77 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_acquire_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27699,53 +29483,77 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28088,53 +29896,77 @@ define amdgpu_kernel void @private_wavefront_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28477,53 +30309,77 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28866,53 +30722,77 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29255,53 +31135,77 @@ define amdgpu_kernel void @private_wavefront_one_as_monotonic_seq_cst_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29644,53 +31548,77 @@ define amdgpu_kernel void @private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30033,53 +31961,77 @@ define amdgpu_kernel void @private_wavefront_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30422,53 +32374,77 @@ define amdgpu_kernel void @private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30811,53 +32787,77 @@ define amdgpu_kernel void @private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_wavefront_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30868,3 +32868,5 @@ entry:
}
attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
index c6e1184880df8..9773250e76db1 100644
--- a/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
+++ b/llvm/test/CodeGen/AMDGPU/memory-legalizer-private-workgroup.ll
@@ -226,44 +226,62 @@ define amdgpu_kernel void @private_workgroup_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") unordered, align 4
@@ -482,44 +500,62 @@ define amdgpu_kernel void @private_workgroup_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") monotonic, align 4
@@ -738,44 +774,62 @@ define amdgpu_kernel void @private_workgroup_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") acquire, align 4
@@ -994,46 +1048,64 @@ define amdgpu_kernel void @private_workgroup_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup") seq_cst, align 4
@@ -1236,43 +1308,60 @@ define amdgpu_kernel void @private_workgroup_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") unordered, align 4
@@ -1474,43 +1563,60 @@ define amdgpu_kernel void @private_workgroup_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") monotonic, align 4
@@ -1712,45 +1818,62 @@ define amdgpu_kernel void @private_workgroup_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") release, align 4
@@ -1952,45 +2075,62 @@ define amdgpu_kernel void @private_workgroup_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup") seq_cst, align 4
@@ -2225,44 +2365,63 @@ define amdgpu_kernel void @private_workgroup_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") monotonic
@@ -2497,45 +2656,64 @@ define amdgpu_kernel void @private_workgroup_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acquire
@@ -2770,46 +2948,65 @@ define amdgpu_kernel void @private_workgroup_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") release
@@ -3044,47 +3241,66 @@ define amdgpu_kernel void @private_workgroup_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acq_rel
@@ -3319,47 +3535,66 @@ define amdgpu_kernel void @private_workgroup_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") seq_cst
@@ -3615,45 +3850,65 @@ define amdgpu_kernel void @private_workgroup_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acquire
@@ -3910,47 +4165,67 @@ define amdgpu_kernel void @private_workgroup_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") acq_rel
@@ -4207,47 +4482,67 @@ define amdgpu_kernel void @private_workgroup_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup") seq_cst
@@ -4560,52 +4855,75 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -4918,53 +5236,76 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5277,54 +5618,77 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5637,55 +6001,78 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -5998,55 +6385,78 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6359,53 +6769,76 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -6718,53 +7151,76 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7077,55 +7533,78 @@ define amdgpu_kernel void @private_workgroup_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7438,55 +7917,78 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -7799,55 +8301,78 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8160,55 +8685,78 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8521,55 +9069,78 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -8882,55 +9453,78 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9243,55 +9837,78 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9604,55 +10221,78 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt_dscnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -9993,53 +10633,77 @@ define amdgpu_kernel void @private_workgroup_monotonic_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10382,53 +11046,77 @@ define amdgpu_kernel void @private_workgroup_acquire_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -10771,55 +11459,79 @@ define amdgpu_kernel void @private_workgroup_release_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11162,55 +11874,79 @@ define amdgpu_kernel void @private_workgroup_acq_rel_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11553,55 +12289,79 @@ define amdgpu_kernel void @private_workgroup_seq_cst_monotonic_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -11944,53 +12704,77 @@ define amdgpu_kernel void @private_workgroup_monotonic_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12333,53 +13117,77 @@ define amdgpu_kernel void @private_workgroup_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -12722,55 +13530,79 @@ define amdgpu_kernel void @private_workgroup_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13113,55 +13945,79 @@ define amdgpu_kernel void @private_workgroup_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13504,55 +14360,79 @@ define amdgpu_kernel void @private_workgroup_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -13895,55 +14775,79 @@ define amdgpu_kernel void @private_workgroup_monotonic_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14286,55 +15190,79 @@ define amdgpu_kernel void @private_workgroup_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -14677,55 +15605,79 @@ define amdgpu_kernel void @private_workgroup_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15068,55 +16020,79 @@ define amdgpu_kernel void @private_workgroup_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15459,55 +16435,79 @@ define amdgpu_kernel void @private_workgroup_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -15728,44 +16728,62 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_unordered_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_unordered_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_unordered_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") unordered, align 4
@@ -15984,44 +17002,62 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") monotonic, align 4
@@ -16240,44 +17276,62 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") acquire, align 4
@@ -16496,46 +17550,64 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_load(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_load:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: flat_load_b32 v0, v[0:1]
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_load:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s1
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_load:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: flat_load_b32 v0, v[0:1]
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %in, ptr addrspace(5) %out) #0 {
entry:
%val = load atomic i32, ptr addrspace(5) %in syncscope("workgroup-one-as") seq_cst, align 4
@@ -16738,43 +17810,60 @@ define amdgpu_kernel void @private_workgroup_one_as_unordered_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_unordered_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_unordered_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_unordered_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") unordered, align 4
@@ -16976,43 +18065,60 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") monotonic, align 4
@@ -17214,45 +18320,62 @@ define amdgpu_kernel void @private_workgroup_one_as_release_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") release, align 4
@@ -17454,45 +18577,62 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_store(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_store:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_store_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_store:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_store:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_store_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
i32 %in, ptr addrspace(5) %out) #0 {
entry:
store atomic i32 %in, ptr addrspace(5) %out syncscope("workgroup-one-as") seq_cst, align 4
@@ -17727,44 +18867,63 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") monotonic
@@ -17999,45 +19158,64 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acquire
@@ -18272,46 +19450,65 @@ define amdgpu_kernel void @private_workgroup_one_as_release_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") release
@@ -18546,47 +19743,66 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acq_rel
@@ -18821,47 +20037,66 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s1, s2
-; GFX1250-NEXT: s_cselect_b32 s2, -1, 0
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s1
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s1, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s1, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v2, s0
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v[0:1], v2
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s1, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s2, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s1
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s1, v2, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s1, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s1, v0, s2
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s0
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v[0:1], v2
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") seq_cst
@@ -19117,45 +20352,65 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acquire
@@ -19412,47 +20667,67 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") acq_rel
@@ -19709,47 +20984,67 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_ret_atomicrmw(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_ret_atomicrmw:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_mov_b32 s2, -1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_cmp_lg_u32 s0, s2
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b32 s2, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
-; GFX1250-NEXT: s_mov_b32 s2, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s2, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s0
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0
+; GFX1250-NOGAS-NEXT: v_mov_b32_e32 v1, s1
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_ret_atomicrmw:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s2, -1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s0, s2
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s2
+; GFX1250-GAS-NEXT: s_mov_b32 s2, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s2, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_swap_b32 v0, v[0:1], v2 th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in) #0 {
entry:
%val = atomicrmw volatile xchg ptr addrspace(5) %out, i32 %in syncscope("workgroup-one-as") seq_cst
@@ -20062,52 +21357,75 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20420,53 +21738,76 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -20779,54 +22120,77 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21139,55 +22503,78 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21500,55 +22887,78 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_monotonic_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -21861,53 +23271,76 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22220,53 +23653,76 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22579,55 +24035,78 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -22940,55 +24419,78 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23301,55 +24803,78 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_acquire_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_acquire_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -23662,55 +25187,78 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24023,55 +25571,78 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24384,55 +25955,78 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -24745,55 +26339,78 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25106,55 +26723,78 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0 offset:16
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_seq_cst_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_add_co_i32 s2, s2, s3
-; GFX1250-NEXT: s_mov_b32 s3, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
-; GFX1250-NEXT: s_mov_b32 s3, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s3, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s2
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s3, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s2, s3
-; GFX1250-NEXT: s_cselect_b32 s3, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[4:5], 0
-; GFX1250-NEXT: s_mov_b32 s2, s5
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s2, s4
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s1
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: v_mov_b32_e32 v4, s0
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v0, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_add_co_i32 s2, s2, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s3, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s3, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s2, s3
+; GFX1250-GAS-NEXT: s_cselect_b32 s3, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s5
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s2, v2, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s2, s4
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s2, v0, s3
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s1
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s0
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v[0:1], v[2:3]
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25495,53 +27135,77 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_monotonic_ret_cmpx
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -25884,53 +27548,77 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26273,55 +27961,79 @@ define amdgpu_kernel void @private_workgroup_one_as_release_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -26664,55 +28376,79 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27055,55 +28791,79 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_monotonic_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_monotonic_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27446,53 +29206,77 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_acquire_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -27835,53 +29619,77 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28224,55 +30032,79 @@ define amdgpu_kernel void @private_workgroup_one_as_release_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -28615,55 +30447,79 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29006,55 +30862,79 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_acquire_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29397,55 +31277,79 @@ define amdgpu_kernel void @private_workgroup_one_as_monotonic_seq_cst_ret_cmpxch
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_monotonic_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -29788,55 +31692,79 @@ define amdgpu_kernel void @private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acquire_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30179,55 +32107,79 @@ define amdgpu_kernel void @private_workgroup_one_as_release_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_release_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_release_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30570,55 +32522,79 @@ define amdgpu_kernel void @private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_acq_rel_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -30961,55 +32937,79 @@ define amdgpu_kernel void @private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg(
; GFX12-CU-NEXT: scratch_store_b32 off, v0, s0
; GFX12-CU-NEXT: s_endpgm
;
-; GFX1250-LABEL: private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg:
-; GFX1250: ; %bb.0: ; %entry
-; GFX1250-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
-; GFX1250-NEXT: s_mov_b64 s[64:65], 0
-; GFX1250-NEXT: v_nop
-; GFX1250-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
-; GFX1250-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
-; GFX1250-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
-; GFX1250-NEXT: s_mov_b32 s3, 16
-; GFX1250-NEXT: s_wait_kmcnt 0x0
-; GFX1250-NEXT: s_add_co_i32 s3, s0, s3
-; GFX1250-NEXT: s_mov_b32 s4, 0
-; GFX1250-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
-; GFX1250-NEXT: s_mov_b32 s4, 20
-; GFX1250-NEXT: v_lshlrev_b32_e64 v2, s4, v0
-; GFX1250-NEXT: v_mov_b32_e32 v0, s3
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
-; GFX1250-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
-; GFX1250-NEXT: v_mov_b32_e32 v2, v1
-; GFX1250-NEXT: s_mov_b32 s4, -1
-; GFX1250-NEXT: s_cmp_lg_u32 s3, s4
-; GFX1250-NEXT: s_cselect_b32 s4, -1, 0
-; GFX1250-NEXT: s_mov_b64 s[6:7], 0
-; GFX1250-NEXT: s_mov_b32 s3, s7
-; GFX1250-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: s_mov_b32 s3, s6
-; GFX1250-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
-; GFX1250-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v1, v2
-; GFX1250-NEXT: v_mov_b32_e32 v2, s2
-; GFX1250-NEXT: v_mov_b32_e32 v4, s1
-; GFX1250-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
-; GFX1250-NEXT: v_mov_b32_e32 v3, v4
-; GFX1250-NEXT: s_wait_loadcnt 0x0
-; GFX1250-NEXT: s_wait_storecnt 0x0
-; GFX1250-NEXT: s_wait_xcnt 0x0
-; GFX1250-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
-; GFX1250-NEXT: s_wait_loadcnt_dscnt 0x0
-; GFX1250-NEXT: scratch_store_b32 off, v0, s0
-; GFX1250-NEXT: s_endpgm
+; GFX1250-NOGAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-NOGAS: ; %bb.0: ; %entry
+; GFX1250-NOGAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-NOGAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-NOGAS-NEXT: v_nop
+; GFX1250-NOGAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s1, s[4:5], 0x4 nv
+; GFX1250-NOGAS-NEXT: s_load_b32 s2, s[4:5], 0x8 nv
+; GFX1250-NOGAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-NOGAS-NEXT: scratch_load_b32 v0, off, s0 offset:16
+; GFX1250-NOGAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-NOGAS-NEXT: v_cmp_eq_u32_e64 s2, v0, s2
+; GFX1250-NOGAS-NEXT: v_cndmask_b32_e64 v1, v0, s1, s2
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v1, s0 offset:16
+; GFX1250-NOGAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-NOGAS-NEXT: s_endpgm
+;
+; GFX1250-GAS-LABEL: private_workgroup_one_as_seq_cst_seq_cst_ret_cmpxchg:
+; GFX1250-GAS: ; %bb.0: ; %entry
+; GFX1250-GAS-NEXT: s_setreg_imm32_b32 hwreg(HW_REG_WAVE_MODE, 25, 1), 1 ; msbs: dst=0 src0=0 src1=0 src2=0
+; GFX1250-GAS-NEXT: s_mov_b64 s[64:65], 0
+; GFX1250-GAS-NEXT: v_nop
+; GFX1250-GAS-NEXT: global_prefetch_b8 v0, s[64:65] scope:SCOPE_SE
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_load_b32 s0, s[4:5], 0x0 nv
+; GFX1250-GAS-NEXT: s_load_b32 s2, s[4:5], 0x4 nv
+; GFX1250-GAS-NEXT: s_load_b32 s1, s[4:5], 0x8 nv
+; GFX1250-GAS-NEXT: s_mov_b32 s3, 16
+; GFX1250-GAS-NEXT: s_wait_kmcnt 0x0
+; GFX1250-GAS-NEXT: s_add_co_i32 s3, s0, s3
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 0
+; GFX1250-GAS-NEXT: v_mbcnt_lo_u32_b32 v0, -1, s4
+; GFX1250-GAS-NEXT: s_mov_b32 s4, 20
+; GFX1250-GAS-NEXT: v_lshlrev_b32_e64 v2, s4, v0
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v0, s3
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: s_mov_b64 s[4:5], src_flat_scratch_base_lo
+; GFX1250-GAS-NEXT: v_add_nc_u64_e64 v[0:1], v[0:1], s[4:5]
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, v1
+; GFX1250-GAS-NEXT: s_mov_b32 s4, -1
+; GFX1250-GAS-NEXT: s_cmp_lg_u32 s3, s4
+; GFX1250-GAS-NEXT: s_cselect_b32 s4, -1, 0
+; GFX1250-GAS-NEXT: s_mov_b64 s[6:7], 0
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s7
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v2, s3, v2, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 killed $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: s_mov_b32 s3, s6
+; GFX1250-GAS-NEXT: v_cndmask_b32_e64 v0, s3, v0, s4
+; GFX1250-GAS-NEXT: ; kill: def $vgpr0 killed $vgpr0 def $vgpr0_vgpr1 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v1, v2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v2, s2
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v4, s1
+; GFX1250-GAS-NEXT: ; kill: def $vgpr2 killed $vgpr2 def $vgpr2_vgpr3 killed $exec
+; GFX1250-GAS-NEXT: v_mov_b32_e32 v3, v4
+; GFX1250-GAS-NEXT: s_wait_loadcnt 0x0
+; GFX1250-GAS-NEXT: s_wait_storecnt 0x0
+; GFX1250-GAS-NEXT: s_wait_xcnt 0x0
+; GFX1250-GAS-NEXT: flat_atomic_cmpswap_b32 v0, v[0:1], v[2:3] th:TH_ATOMIC_RETURN
+; GFX1250-GAS-NEXT: s_wait_loadcnt_dscnt 0x0
+; GFX1250-GAS-NEXT: scratch_store_b32 off, v0, s0
+; GFX1250-GAS-NEXT: s_endpgm
ptr addrspace(5) %out, i32 %in, i32 %old) #0 {
entry:
%gep = getelementptr i32, ptr addrspace(5) %out, i32 4
@@ -31020,3 +33020,5 @@ entry:
}
attributes #0 = { nounwind }
+;; NOTE: These prefixes are unused and the list is autogenerated. Do not add tests below this line:
+; GFX1250: {{.*}}
diff --git a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
index c77e06a641ad1..feb8ab4fc5574 100644
--- a/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
+++ b/llvm/test/Transforms/AtomicExpand/AMDGPU/expand-atomic-private-gas.ll
@@ -9,11 +9,16 @@ define void @system_atomic_store_unordered_float(ptr addrspace(5) %addr, float %
; GFX1200-NEXT: store float [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @system_atomic_store_unordered_float(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[VAL:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic float [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @system_atomic_store_unordered_float(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[VAL:%.*]]) {
+; GFX1250-NOGAS-NEXT: store float [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @system_atomic_store_unordered_float(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[VAL:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic float [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic float %val, ptr addrspace(5) %addr unordered, align 4
ret void
@@ -25,11 +30,16 @@ define void @system_atomic_store_unordered_i32(ptr addrspace(5) %addr, i32 %val)
; GFX1200-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @system_atomic_store_unordered_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @system_atomic_store_unordered_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
+; GFX1250-NOGAS-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @system_atomic_store_unordered_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] unordered, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic i32 %val, ptr addrspace(5) %addr unordered, align 4
ret void
@@ -41,11 +51,16 @@ define void @system_atomic_store_release_i32(ptr addrspace(5) %addr, i32 %val) {
; GFX1200-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @system_atomic_store_release_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] release, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @system_atomic_store_release_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
+; GFX1250-NOGAS-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @system_atomic_store_release_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] release, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic i32 %val, ptr addrspace(5) %addr release, align 4
ret void
@@ -57,11 +72,16 @@ define void @workgroup_atomic_store_release_i32(ptr addrspace(5) %addr, i32 %val
; GFX1200-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret void
;
-; GFX1250-LABEL: define void @workgroup_atomic_store_release_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] syncscope("workgroup") release, align 4
-; GFX1250-NEXT: ret void
+; GFX1250-NOGAS-LABEL: define void @workgroup_atomic_store_release_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
+; GFX1250-NOGAS-NEXT: store i32 [[VAL]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret void
+;
+; GFX1250-GAS-LABEL: define void @workgroup_atomic_store_release_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[VAL:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: store atomic i32 [[VAL]], ptr [[SCRATCH_ASCAST]] syncscope("workgroup") release, align 4
+; GFX1250-GAS-NEXT: ret void
;
store atomic i32 %val, ptr addrspace(5) %addr syncscope("workgroup") release, align 4
ret void
@@ -73,11 +93,16 @@ define float @system_atomic_load_unordered_float(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load float, ptr addrspace(5) [[ADDR]], align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
; GFX1200-NEXT: ret float [[VAL]]
;
-; GFX1250-LABEL: define float @system_atomic_load_unordered_float(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic float, ptr [[SCRATCH_ASCAST]] unordered, align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
-; GFX1250-NEXT: ret float [[VAL]]
+; GFX1250-NOGAS-LABEL: define float @system_atomic_load_unordered_float(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load float, ptr addrspace(5) [[ADDR]], align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
+; GFX1250-NOGAS-NEXT: ret float [[VAL]]
+;
+; GFX1250-GAS-LABEL: define float @system_atomic_load_unordered_float(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic float, ptr [[SCRATCH_ASCAST]] unordered, align 4, !invariant.load [[META0:![0-9]+]], !nontemporal [[META1:![0-9]+]]
+; GFX1250-GAS-NEXT: ret float [[VAL]]
;
%val = load atomic float, ptr addrspace(5) %addr unordered, align 4, !invariant.load !1, !nontemporal !0
ret float %val
@@ -89,11 +114,16 @@ define i32 @system_atomic_load_unordered_i32(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[VAL]]
;
-; GFX1250-LABEL: define i32 @system_atomic_load_unordered_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] unordered, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomic_load_unordered_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[VAL]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomic_load_unordered_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] unordered, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = load atomic i32, ptr addrspace(5) %addr unordered, align 4
ret i32 %val
@@ -105,11 +135,16 @@ define i32 @system_atomic_load_acquire_i32(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[VAL]]
;
-; GFX1250-LABEL: define i32 @system_atomic_load_acquire_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] acquire, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomic_load_acquire_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[VAL]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomic_load_acquire_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] acquire, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = load atomic i32, ptr addrspace(5) %addr acquire, align 4
ret i32 %val
@@ -121,11 +156,16 @@ define i32 @workgroup_atomic_load_acquire_i32(ptr addrspace(5) %addr) {
; GFX1200-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[VAL]]
;
-; GFX1250-LABEL: define i32 @workgroup_atomic_load_acquire_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] syncscope("workgroup") acquire, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @workgroup_atomic_load_acquire_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[VAL:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[VAL]]
+;
+; GFX1250-GAS-LABEL: define i32 @workgroup_atomic_load_acquire_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = load atomic i32, ptr [[SCRATCH_ASCAST]] syncscope("workgroup") acquire, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = load atomic i32, ptr addrspace(5) %addr syncscope("workgroup") acquire, align 4
ret i32 %val
@@ -143,12 +183,23 @@ define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(ptr addrspace(5) %addr, i3
; GFX1200-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[TMP5]], 0
; GFX1200-NEXT: ret i32 [[RES]]
;
-; GFX1250-LABEL: define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[OLD:%.*]], i32 [[IN:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = cmpxchg volatile ptr [[SCRATCH_ASCAST]], i32 [[OLD]], i32 [[IN]] acq_rel acquire, align 4, !nontemporal [[META1]]
-; GFX1250-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[VAL]], 0
-; GFX1250-NEXT: ret i32 [[RES]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[OLD:%.*]], i32 [[IN:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: [[TMP2:%.*]] = icmp eq i32 [[TMP1]], [[OLD]]
+; GFX1250-NOGAS-NEXT: [[TMP3:%.*]] = select i1 [[TMP2]], i32 [[IN]], i32 [[TMP1]]
+; GFX1250-NOGAS-NEXT: store i32 [[TMP3]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: [[TMP4:%.*]] = insertvalue { i32, i1 } poison, i32 [[TMP1]], 0
+; GFX1250-NOGAS-NEXT: [[TMP5:%.*]] = insertvalue { i32, i1 } [[TMP4]], i1 [[TMP2]], 1
+; GFX1250-NOGAS-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[TMP5]], 0
+; GFX1250-NOGAS-NEXT: ret i32 [[RES]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomic_cmpxchg_acq_rel_acquire_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[OLD:%.*]], i32 [[IN:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = cmpxchg volatile ptr [[SCRATCH_ASCAST]], i32 [[OLD]], i32 [[IN]] acq_rel acquire, align 4, !nontemporal [[META1]]
+; GFX1250-GAS-NEXT: [[RES:%.*]] = extractvalue { i32, i1 } [[VAL]], 0
+; GFX1250-GAS-NEXT: ret i32 [[RES]]
;
%val = cmpxchg volatile ptr addrspace(5) %addr, i32 %old, i32 %in acq_rel acquire, !nontemporal !0
%res = extractvalue { i32, i1 } %val, 0
@@ -162,11 +213,17 @@ define i32 @system_atomicrmw_xchg_acq_rel_i32(ptr addrspace(5) %addr, i32 %in) {
; GFX1200-NEXT: store i32 [[IN]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret i32 [[TMP1]]
;
-; GFX1250-LABEL: define i32 @system_atomicrmw_xchg_acq_rel_i32(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[IN:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i32 [[IN]] acq_rel, align 4
-; GFX1250-NEXT: ret i32 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i32 @system_atomicrmw_xchg_acq_rel_i32(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[IN:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load i32, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: store i32 [[IN]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret i32 [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define i32 @system_atomicrmw_xchg_acq_rel_i32(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i32 [[IN:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i32 [[IN]] acq_rel, align 4
+; GFX1250-GAS-NEXT: ret i32 [[VAL]]
;
%val = atomicrmw volatile xchg ptr addrspace(5) %addr, i32 %in acq_rel
ret i32 %val
@@ -179,11 +236,17 @@ define i16 @system_atomicrmw_xchg_acq_rel_i16(ptr addrspace(5) %addr, i16 %in) {
; GFX1200-NEXT: store i16 [[IN]], ptr addrspace(5) [[ADDR]], align 2
; GFX1200-NEXT: ret i16 [[TMP1]]
;
-; GFX1250-LABEL: define i16 @system_atomicrmw_xchg_acq_rel_i16(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], i16 [[IN:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i16 [[IN]] acq_rel, align 2
-; GFX1250-NEXT: ret i16 [[VAL]]
+; GFX1250-NOGAS-LABEL: define i16 @system_atomicrmw_xchg_acq_rel_i16(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i16 [[IN:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load i16, ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: store i16 [[IN]], ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: ret i16 [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define i16 @system_atomicrmw_xchg_acq_rel_i16(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], i16 [[IN:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile xchg ptr [[SCRATCH_ASCAST]], i16 [[IN]] acq_rel, align 2
+; GFX1250-GAS-NEXT: ret i16 [[VAL]]
;
%val = atomicrmw volatile xchg ptr addrspace(5) %addr, i16 %in acq_rel
ret i16 %val
@@ -197,11 +260,18 @@ define half @system_atomicrmw_fmax_acq_rel_half(ptr addrspace(5) %addr, half %in
; GFX1200-NEXT: store half [[TMP2]], ptr addrspace(5) [[ADDR]], align 2
; GFX1200-NEXT: ret half [[TMP1]]
;
-; GFX1250-LABEL: define half @system_atomicrmw_fmax_acq_rel_half(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], half [[IN:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile fmax ptr [[SCRATCH_ASCAST]], half [[IN]] acq_rel, align 2
-; GFX1250-NEXT: ret half [[VAL]]
+; GFX1250-NOGAS-LABEL: define half @system_atomicrmw_fmax_acq_rel_half(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], half [[IN:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load half, ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: [[TMP2:%.*]] = call half @llvm.maxnum.f16(half [[TMP1]], half [[IN]])
+; GFX1250-NOGAS-NEXT: store half [[TMP2]], ptr addrspace(5) [[ADDR]], align 2
+; GFX1250-NOGAS-NEXT: ret half [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define half @system_atomicrmw_fmax_acq_rel_half(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], half [[IN:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile fmax ptr [[SCRATCH_ASCAST]], half [[IN]] acq_rel, align 2
+; GFX1250-GAS-NEXT: ret half [[VAL]]
;
%val = atomicrmw volatile fmax ptr addrspace(5) %addr, half %in acq_rel
ret half %val
@@ -215,11 +285,18 @@ define float @system_atomicrmw_fminimum_acq_rel_float(ptr addrspace(5) %addr, fl
; GFX1200-NEXT: store float [[TMP2]], ptr addrspace(5) [[ADDR]], align 4
; GFX1200-NEXT: ret float [[TMP1]]
;
-; GFX1250-LABEL: define float @system_atomicrmw_fminimum_acq_rel_float(
-; GFX1250-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[IN:%.*]]) {
-; GFX1250-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
-; GFX1250-NEXT: [[VAL:%.*]] = atomicrmw volatile fminimum ptr [[SCRATCH_ASCAST]], float [[IN]] acq_rel, align 4, !nontemporal [[META1]]
-; GFX1250-NEXT: ret float [[VAL]]
+; GFX1250-NOGAS-LABEL: define float @system_atomicrmw_fminimum_acq_rel_float(
+; GFX1250-NOGAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[IN:%.*]]) {
+; GFX1250-NOGAS-NEXT: [[TMP1:%.*]] = load float, ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: [[TMP2:%.*]] = call float @llvm.minimum.f32(float [[TMP1]], float [[IN]])
+; GFX1250-NOGAS-NEXT: store float [[TMP2]], ptr addrspace(5) [[ADDR]], align 4
+; GFX1250-NOGAS-NEXT: ret float [[TMP1]]
+;
+; GFX1250-GAS-LABEL: define float @system_atomicrmw_fminimum_acq_rel_float(
+; GFX1250-GAS-SAME: ptr addrspace(5) [[ADDR:%.*]], float [[IN:%.*]]) {
+; GFX1250-GAS-NEXT: [[SCRATCH_ASCAST:%.*]] = addrspacecast ptr addrspace(5) [[ADDR]] to ptr
+; GFX1250-GAS-NEXT: [[VAL:%.*]] = atomicrmw volatile fminimum ptr [[SCRATCH_ASCAST]], float [[IN]] acq_rel, align 4, !nontemporal [[META1]]
+; GFX1250-GAS-NEXT: ret float [[VAL]]
;
%val = atomicrmw volatile fminimum ptr addrspace(5) %addr, float %in acq_rel, !nontemporal !0
ret float %val
More information about the llvm-commits
mailing list